Kienan Stewart
33fa4bd266
Update container compose to connect watcher to the selenium driver
2022-09-25 20:39:29 -04:00
Kienan Stewart
2a25b41baf
Load default driver url from environment variable if it is set
2022-09-25 20:39:06 -04:00
Kienan Stewart
cad05a6609
Recover last value from rss channel when creating thread jobs
2022-09-25 19:49:22 -04:00
Kienan Stewart
4b36b75d9f
Add first pass at using inotify to watch the job directory for changes
...
There are number of cases which currently aren't handled:
* moving a file out of or into the directory
* a file being touched: should that be used to reset that last_run
value?
* when jobs are removed, it is regardless of thread state so there
may be children that never get joined
2022-09-25 18:36:40 -04:00
Kienan Stewart
518f0b284f
Add method to check job extension on path
2022-09-25 18:12:20 -04:00
Kienan Stewart
da4e79463f
Load jobs from job files in job directory
2022-09-25 17:50:01 -04:00
Kienan Stewart
82bd7e2d5c
Create channel when creating job from file
2022-09-25 17:49:26 -04:00
Kienan Stewart
d2866c3f37
Set source_file on jobs when created from a job file
2022-09-25 17:26:27 -04:00
Kienan Stewart
fef475c8ad
Set default output_file for jobs when loaded from file
2022-09-25 17:16:50 -04:00
Kienan Stewart
38b5d5bb32
Split Job struct into it's own module
2022-09-25 17:11:48 -04:00
Kienan Stewart
514209dae2
Refactor reading configuration file into a separate function
...
This allows for reusing it elsewhere (eg. for job configuration files)
2022-09-25 16:39:25 -04:00
Kienan Stewart
1bbebe76db
WIP: Add container for building & running haunter
2022-09-25 15:12:37 -04:00
Kienan Stewart
ddff59cbac
Add web service to container-compose to serve result files
2022-09-25 08:52:58 -04:00
Kienan Stewart
b33cd1e705
Create rss feeds from updates
2022-09-24 18:41:00 -04:00
Kienan Stewart
20870b6444
Own the memory for url & selector in Job structs
2022-09-24 11:57:07 -04:00
Kienan Stewart
b1dee1f5a8
Fix typo
2022-09-24 11:38:36 -04:00
Kienan Stewart
9cb06f8323
Add a comment to configuration file test
2022-09-24 11:38:11 -04:00
Kienan Stewart
52cb639511
Convert paths from String to PathBuf
2022-09-24 11:36:08 -04:00
Kienan Stewart
222d4c6c18
Verify that the output and job directories exist and are writable
2022-09-24 11:08:45 -04:00
Kienan Stewart
053910c03f
Use driver url from configuration
2022-09-24 10:34:26 -04:00
Kienan Stewart
6c0e022f7a
Test that a value containing '=' is not split more than once
2022-09-24 10:24:00 -04:00
Kienan Stewart
17a9445796
Allow driver_url to be configurable
2022-09-24 10:23:50 -04:00
Kienan Stewart
6fd8bf4304
Switch from chrome driver to firefox driver and increase concurrency
2022-09-24 10:13:51 -04:00
Kienan Stewart
cf5ebbb457
Run least recently run jobs up to max_running_tasks
2022-09-24 10:05:39 -04:00
Kienan Stewart
24baa3a6a6
Load default configuration overrides from files
2022-09-23 22:27:13 -04:00
Kienan Stewart
a6784fdce0
Initial commit
2022-09-23 19:56:04 -04:00
Kienan Stewart