Configurable web automation framework, do you people need it?

Pages: [1]
Author Topic: Configurable web automation framework, do you people need it?  (Read 1279 times)
tomkruiz
Gaining experience
*

Karma: 10
Posts: 43


View Profile

tomkruiz

Configurable web automation framework, do you people need it?
« on: November 04, 2019, 06:08:04 PM »

Hello everyone,

I have recently come to a conclusion that a project I've built, can't show its potential or even see a light of the day properly if it keeps being used by one-man band such as me. So I'd like to make an inquiry which would hopefully result in a clear answer to what steps I should take regarding the project.

I have already written to site owners about this, but I feel like I missed the target audience, so hopefully this way I get things straight and obtain the feedback I actually needed. I owe a thank you to the admin, however, for taking the time to actually read the whole message (which might not be an easy task ;D)

The project
The project I've built is a kind of a configurable web automation engine with a user interface, where the user interface allows one to create so called configurations. The configuration essentially consists of "actions", with the ability to wait for certain conditions before or after each action is executed. The currently supported actions should be enough to scrape data from any kind of web site.

Currently supported actions are, simply put:
  • navigation to a URL;
  • arbitrary script execution;
  • selected web page element click;
  • conditional action, i.e. if a certain condition is met, execute this action, else, execute that action;
  • enumeration action, i.e. find all elements which meet certain criteria and then perform further actions with each of them.

The automation engine is usually plugged into another project which uses the output from it and performs analysis, i.e. the automation engine uses configurations which tell the engine how to collect odds from various sportbooks, and then the project which the engine is plugged into performs the analysis of collected odds and finds arbs, values bets or w/e, or just stores the collected data for data warehousing needs, or.... you get the idea.

Here is an example of how the interface to view the configuration actions look like at the moment (follow the link so we don't overstretch this post, it's already stretched enough as it is): https://drive.google.com/file/d/1QeF3A3-jyaQEiMLffGo2MRhK-pp-BTOO/view
The image doesn't show it, but each action depicted above has it's own interface (depending on the action type) which allows one to edit the action, set pre-execute or post-execute wait condition, and all things relevant to that action.

Bear with me, as the UI is currently localised to my language. But I've made sure it's easy to localise to any language required (for those of you who are tech savvy, i18n standard is used on the front-end application).

This image represents, what once was, a valid and tested configuration to obtain odds for certain markets for all the available football games from bet365. They have changed some of their layout in the meanwhile, which is understandable since this configuration has last been updated more than 5 months ago.

The idea
The idea is to create a service which would allow users to create their own automation configurations. Users would then be able to create and download a portable executable file based on the configuration they created. Such generated executable file would be able to execute their created automation configuration from their computers, or from whichever machine they find fitting for it.

That executable file would then be able to provide an endpoint which users could use to read the data produced by the automation, or write the data to some configured output, such as a file, or both. Obviously, this service would target people who know how web pages work, because in order to create a configuration, one would either have to know about web page layout and elements, or pay someone else to create a configuration for them.

The service could also have a function to host some of the user-created configurations as well, execute them and provide an endpoint for the requesting user to be able to read the data.

This community has given me knowledge about the vast world of smart betting, so I'd like to give something back. Therefore I'm thinking, after the project is ready, to offer it, as I have imagined it, to arbusers community members for free.

But it's not quite there, just yet. First I need to see whether people actually need this in the form I described above. So be free to comment with your ideas or suggestions. After I gather the feedback, hopefully positive, I will take time and shape the service to the form that we agreed here in this thread. That might also take some time as I'm a one-man band, but I'd try to wrap things up as soon as possible, after all, in smart betting, I'm not sure time is our ally.

And thanks for reading.
« Last Edit: November 04, 2019, 06:11:37 PM by tomkruiz » Logged
VidaBlue
Has experience
**

Karma: 42
Posts: 180


View Profile

VidaBlue

Re: Configurable web automation framework, do you people need it?
« Reply #1 on: November 05, 2019, 08:51:54 AM »

It is very generous of you tomkruiz, to offer something for free. Thank you.

I am trying to understand what this tool does and where it would fit into a betting data regime, so this answer is a search for clarification.

I will present my process, when developing and executing automated odds comparison from bookie X:
  • Surveying the site configuration of Bookie X
  • Writing the code to gather (web scrape) data from Bookie X
  • Execute this code and similar codes (or APIs) for other bookmakers
  • Execute fuzzy matching on Bookie X events with the other bookmaker events
  • Execute odds comparison based upon Bookie X events
  • Execute presentation of sorted lists of value bets for Bookie X
I am not a professional programmer and I work alone, so I don’t know if this is the best practice to build these things, but it seems to work for my purpose. I have spent much time on development though and additional time with maintenance when site scripts have changed.

I don’t see clearly what your project does, but based upon the description of my own regime, I am guessing:

It is a tool where the user configures how web scraping is done in an easier manner than having to hard code this. In my case, if I modify my project to use these “configurations” instead of my endless cumbersome lines of web scraping code, I will have an easier and faster approach to the 2nd bullet above.

If this is in line with what the tool actually does, I would be interested.
Logged
tomkruiz
Gaining experience
*

Karma: 10
Posts: 43


View Profile

tomkruiz

Re: Configurable web automation framework, do you people need it?
« Reply #2 on: November 05, 2019, 09:33:51 AM »

It is a tool where the user configures how web scraping is done in an easier manner than having to hard code this. In my case, if I modify my project to use these “configurations” instead of my endless cumbersome lines of web scraping code, I will have an easier and faster approach to the 2nd bullet above.

Yes, that is the idea. I'm thinking that might be useful for anyone who has their own web scrapper, and especially for anyone who might have plans to create their own web scrapper, as having a tool like this would save them some time. The goal is to have a product with a polished user experience, so writing all those automation steps with XPath and CSS selectors doesn't seem that cumbersome.
Logged
galt88
Gaining experience
*

Karma: 6
Posts: 42



View Profile

galt88

Re: Configurable web automation framework, do you people need it?
« Reply #3 on: November 05, 2019, 03:04:11 PM »

Hi, i think your project would be very useful for a lot of users from this forum.If you will finish the product I would like to collaborate with you about making a bot.
Logged
esi
Gaining experience
*

Karma: 3
Posts: 32


View Profile

esi

Re: Configurable web automation framework, do you people need it?
« Reply #4 on: November 05, 2019, 06:44:24 PM »

That's nice, it would be good if the bot automates chromium and doesn't use a web driver since web drivers can be easily detected
Looking forward to seeing the project
Logged
tomkruiz
Gaining experience
*

Karma: 10
Posts: 43


View Profile

tomkruiz

Re: Configurable web automation framework, do you people need it?
« Reply #5 on: November 05, 2019, 09:01:03 PM »

Thank you for the feedback guys.

@esi, the engine indeed uses the web driver, however there is also a mechanism in place to prevent detection. Both the engine and the said mechanism are fully functional. However, the way I see it, there are the following tasks ahead of me:

  • Implementing the service so it's user-centric, which would allow people to log in to it and create their configurations. This task will require some database schema modifications, some service refactorings and some changes to the front-end. Overall, might take some 16 to 24 work hours.
  • Implementing the ability to generate portable executables with embedded configurations that users have created, which when executed, parse the embedded configuration. This one needs to be done from scratch, so it'll likely take me some 16 work hours only to investigate and figure out how to create such portable executables, and then probably some 24 work hours more to figure out and implement the endpoint which users will be able to listen to, in order to obtain the return values from the portable service contained in the executable. It needs to provide a way to obtain the data as it's been scraped, would be useless if users would need to wait for the scraping process to finish.

I think this is the best-case scenario. But unfortunately, there is not much time I can allocate to work on these tasks, so they will likely take more than 40 hours to complete. On the other hand, I'm receiving some good vibes from your feedback, which might speed things a little. But please, bear with me, I am a busy man :)
Logged
Wolfie
Pro
****

Karma: 23
Posts: 372



View Profile

Wolfie

Re: Configurable web automation framework, do you people need it?
« Reply #6 on: November 06, 2019, 10:17:55 AM »

I am also interesed in this.
Will web scraping work for all bookies ?
Will you turn it into a product where even non programmers can profit or programming skills will still be needed for step 2 ?
Logged
tomkruiz
Gaining experience
*

Karma: 10
Posts: 43


View Profile

tomkruiz

Re: Configurable web automation framework, do you people need it?
« Reply #7 on: November 06, 2019, 10:38:36 AM »

Hey @Wolfie,

currently, the engine supports only those websites which are accessible to public, i.e. the login is not required in order to see the real information. If proven successful, the engine will feature the login support too, in the future.

About whether the non-programmers will be able to profit from this, I suggest you take a look at the screenshot I provided in the OP. If a person doesn't know programming, but understands XPath and CSS selectors depicted in that screenshot, they will be able to use this product.

At the end of the day, this product is supposed to yield some scrapped data, such as games, markets and odds. What users do with that output, i.e. whether they use their programming skills or parse it manually, is out of this project's scope.
Logged
barbero
Pro
****

Karma: 25
Posts: 364


View Profile

barbero

Re: Configurable web automation framework, do you people need it?
« Reply #8 on: November 07, 2019, 10:02:45 AM »

This engine will allow many of us to try to take it a step further! I can think of some good uses :)

I would love to try it if given the chance, thank you very much for your contribution tom. I must also say it's not only a very nice idea but especially very impressive (at least to me).
Logged
Wolfie
Pro
****

Karma: 23
Posts: 372



View Profile

Wolfie

Re: Configurable web automation framework, do you people need it?
« Reply #9 on: November 07, 2019, 10:10:29 AM »

Thanks for taking your time and sharing this @Tomkruiz
Will sustainability be a problem ? Since sites change from time to time. Or does this tool work even after changes ? If yes it would be great.
Do you actually use this for some of your projects ?
Logged
tomkruiz
Gaining experience
*

Karma: 10
Posts: 43


View Profile

tomkruiz

Re: Configurable web automation framework, do you people need it?
« Reply #10 on: November 07, 2019, 10:54:35 AM »

@barbero thanks for the kind words, I look forward to it. However, don't thank me yet, let's wait for it to be released.

@Wolfie, I actually started working on this almost a year ago, with an idea to make my own arb finder and alert service and stop paying for them. Then I started looking into value bets too, so I wanted to use it for that too. Then I realized I'd spent too much time placing bets, so I wanted to fully automate the betting too. Then I realized that I need a configurable engine which should be able to support all that. However, somewhere in May I had some investments in both time and money that I needed to make, so I stopped working on this project in order to focus on projects which gave me the much needed short-term profit. Couple of months later, I even contacted a couple of addresses from the industry in order to try to sell the work which was done so far. However, it seems as I either haven't marketed it good enough, or people simply didn't need it.

So a month ago, I finally found some spare time to start working on it again. Unfortunately, the backlog of tasks towards the intended original goal is too long and it would take some time for it to be finished and it was just a matter of time before a new project popped up which would again drag my attention from this. And then this idea came to me, why wouldn't I create a service with a number of users which will serve as a stress test, to see how well the engine really fares. That real world usage would also allow me to identify priorities in regards to the engine development.

As for the sustainability question, it depends on how well you create your configuration. Some configurations would probably be immune to some small cosmetic site changes, but at the end of the day, if your configuration stops working, it is assumed that you can always login to the service, amend the configuration as per the target site changes and download a new executable.
Logged
Wolfie
Pro
****

Karma: 23
Posts: 372



View Profile

Wolfie

Re: Configurable web automation framework, do you people need it?
« Reply #11 on: November 07, 2019, 11:17:00 AM »

@tomkruiz
I know man this times are very dynamic. You dont know where to put your time. And projects which require too much time when you finish them the situation may be different and the end product not profitable. This makes most of people short time profit oriented. But if you find some long time project which does not expire or at least its elements will still be needed even if the primary idea goes down than you are in the right track. I think this project is one of those. If it works better than most of present scraping options than you may have a winner. I think the community is very interested for a new better, community oriented alert service.
Logged
djordjeno
Has experience
**

Karma: 6
Posts: 104


View Profile

djordjeno

Re: Configurable web automation framework, do you people need it?
« Reply #12 on: November 08, 2019, 12:22:56 PM »

@tomkruiz

Dobra ideja ;)

Do you have some website/webservice where this can be tested?
In which output format you plan to serve parsed data? Probably you have own defined model or it can be defined by transformation?
Logged
tomkruiz
Gaining experience
*

Karma: 10
Posts: 43


View Profile

tomkruiz

Re: Configurable web automation framework, do you people need it?
« Reply #13 on: November 08, 2019, 01:22:06 PM »

@tomkruiz

Dobra ideja ;)

Do you have some website/webservice where this can be tested?
In which output format you plan to serve parsed data? Probably you have own defined model or it can be defined by transformation?
;)

Can't be tested yet, I'm working on it to bring it online soon.
The model can be defined by users. When creating a configuration, users define the model, and then when adding action steps to the configuration, each step can yield some output which can be bound to one or multiple model fields.

Regarding the output format, I am not sure yet. Do you have any suggestion, what would you prefer as an output?
Logged
djordjeno
Has experience
**

Karma: 6
Posts: 104


View Profile

djordjeno

Re: Configurable web automation framework, do you people need it?
« Reply #14 on: November 12, 2019, 11:39:58 AM »

JSON is to today preferred.

For your bussiness part you can try to publish your solution / api via RapidApi platform.
I do not know details but seems to be popular because you can focus on development not to marketing :)


Logged
Pages: [1]
Print