Kienan Stewart
a69fd14ea4
Use text-diff to compare run results
2022-10-08 20:39:55 -04:00
Kienan Stewart
3c21f234d8
Remove new lines in the new & diff values of the update content
2022-10-08 09:06:45 -04:00
Kienan Stewart
277854b596
Escape diff of the content to increase readability
2022-10-08 09:03:22 -04:00
Kienan Stewart
52e66a0372
Don't escape HTML entities in RSS feed output
...
There is a risk I guess that the the value and diff contain </pre>
which would break the sort of HTML content, but the important part
was to convert the ansi escapes codes rather than escape HTML.
The HTML escaping of the ansi_to_html module is also on a partial
escape, and not very robust. It might be worth using something like
htmlize or html_escape which provide full conversions to both
encoding and decoding HTML entities.
2022-10-07 17:31:30 -04:00
Kienan Stewart
44253beea9
Add guid to items created during updates
2022-10-07 16:41:31 -04:00
Kienan Stewart
43e97d4db7
Load defaults from environment variables
2022-10-02 15:37:46 -04:00
Kienan Stewart
2a25b41baf
Load default driver url from environment variable if it is set
2022-09-25 20:39:06 -04:00
Kienan Stewart
cad05a6609
Recover last value from rss channel when creating thread jobs
2022-09-25 19:49:22 -04:00
Kienan Stewart
4b36b75d9f
Add first pass at using inotify to watch the job directory for changes
...
There are number of cases which currently aren't handled:
* moving a file out of or into the directory
* a file being touched: should that be used to reset that last_run
value?
* when jobs are removed, it is regardless of thread state so there
may be children that never get joined
2022-09-25 18:36:40 -04:00
Kienan Stewart
518f0b284f
Add method to check job extension on path
2022-09-25 18:12:20 -04:00
Kienan Stewart
da4e79463f
Load jobs from job files in job directory
2022-09-25 17:50:01 -04:00
Kienan Stewart
82bd7e2d5c
Create channel when creating job from file
2022-09-25 17:49:26 -04:00
Kienan Stewart
d2866c3f37
Set source_file on jobs when created from a job file
2022-09-25 17:26:27 -04:00
Kienan Stewart
fef475c8ad
Set default output_file for jobs when loaded from file
2022-09-25 17:16:50 -04:00
Kienan Stewart
38b5d5bb32
Split Job struct into it's own module
2022-09-25 17:11:48 -04:00
Kienan Stewart
514209dae2
Refactor reading configuration file into a separate function
...
This allows for reusing it elsewhere (eg. for job configuration files)
2022-09-25 16:39:25 -04:00
Kienan Stewart
b33cd1e705
Create rss feeds from updates
2022-09-24 18:41:00 -04:00
Kienan Stewart
20870b6444
Own the memory for url & selector in Job structs
2022-09-24 11:57:07 -04:00
Kienan Stewart
b1dee1f5a8
Fix typo
2022-09-24 11:38:36 -04:00
Kienan Stewart
9cb06f8323
Add a comment to configuration file test
2022-09-24 11:38:11 -04:00
Kienan Stewart
52cb639511
Convert paths from String to PathBuf
2022-09-24 11:36:08 -04:00
Kienan Stewart
222d4c6c18
Verify that the output and job directories exist and are writable
2022-09-24 11:08:45 -04:00
Kienan Stewart
053910c03f
Use driver url from configuration
2022-09-24 10:34:26 -04:00
Kienan Stewart
6c0e022f7a
Test that a value containing '=' is not split more than once
2022-09-24 10:24:00 -04:00
Kienan Stewart
17a9445796
Allow driver_url to be configurable
2022-09-24 10:23:50 -04:00
Kienan Stewart
6fd8bf4304
Switch from chrome driver to firefox driver and increase concurrency
2022-09-24 10:13:51 -04:00
Kienan Stewart
cf5ebbb457
Run least recently run jobs up to max_running_tasks
2022-09-24 10:05:39 -04:00
Kienan Stewart
24baa3a6a6
Load default configuration overrides from files
2022-09-23 22:27:13 -04:00
Kienan Stewart
a6784fdce0
Initial commit
2022-09-23 19:56:04 -04:00
Kienan Stewart