a3b2748479
fix #87 : document image distance hash
2024-04-26 13:42:28 +02:00
8cc5a9e3ed
missed commits
2024-02-12 13:36:29 +01:00
d2bcd7b505
fix #80 : using os.MkdirAll():
...
Recursively create ad dir including output dir. The output dir itself
is not being created separately anymore. That way, no directory will
be created if no ads could be downloaded.
2024-02-12 13:32:25 +01:00
c59c2e2931
fix #81 : add arm64 build support
2024-02-12 13:32:25 +01:00
2288806105
fix #77 : use processed ad dir for duplicate checking, not slug
2024-02-10 15:15:43 +01:00
5a2c277f0e
fix #71 and #73 : add support for outdir template and enhance docs
2024-02-10 14:44:09 +01:00
612ed2aa79
fix #74 : warn if about to write to already visited ad, overwrite if -f
2024-02-10 14:44:09 +01:00
ed78731b3c
check seek error
2024-01-27 17:34:44 +01:00
a84f0e1436
get rid of duplicate bytes.Buffer, use bytes.Reader instead, #39
2024-01-27 17:34:44 +01:00
d8d5be5c7d
fix #58 : add missing dashes to self issue template
2024-01-27 17:34:44 +01:00
bcf920c91e
correct #39 add --ignoreerrors flag
2024-01-27 17:34:44 +01:00
T.v.Dein
14f8c3fd43
Fix/linter ( #66 )
...
* added lint targets
* fix linter errors
* enhance error handling
* !!BREAKING!! rename Id to ID in tpls
2024-01-25 19:04:15 +01:00
9cd1fc0596
behavior changes: UserAgent configurable, test cookies, check errors
2024-01-24 19:22:31 +01:00
8df3ebfa6d
add throttling to image download
2024-01-24 19:22:31 +01:00
de82127223
first step in fixing #49 :
...
fetch cookies from 1st response and use them in subsequent requests.
2024-01-24 19:22:31 +01:00
a79a28f4a1
add contribution guidelines and non-code-of-conduct
2024-01-23 18:01:14 +01:00
95b1172b7f
fix typo
2024-01-23 17:26:06 +01:00
6f3954f1c0
added issue templates and fix make show-versions
2024-01-23 14:20:33 +01:00
a465bbfaa8
Added german README, update docker docs, add latest image tag
2024-01-23 14:20:33 +01:00
ddae6ed8be
bump version (feature update)
2024-01-22 18:43:44 +01:00
8cced1f9e1
Fix #47 : add mock http server for testing w/o hitting prod site
2024-01-22 18:43:44 +01:00
3245438564
update yadu to v0.1.1, fix #46
2024-01-22 18:43:44 +01:00
71c528114f
fix linter error
2024-01-22 18:43:44 +01:00
e2afc1350b
added -f to override d-hash, better debug and error handling
2024-01-22 18:43:44 +01:00
e971070f9f
added image diff hash distance caching to not overwrite similar images
2024-01-22 18:43:44 +01:00
T.v.Dein
8771ec1108
added support to calculate and store the ad expire date ( #43 )
2024-01-19 18:38:35 +01:00
T.v.Dein
1896209b96
use new yadu log handler, +tests, +upd modules, +version
2024-01-19 18:38:35 +01:00
T.v.Dein
3c93c9fce0
added docker image support
...
* added environment variable support
* added docker instructions
* added .env hint
2024-01-19 18:38:35 +01:00
T.v.Dein
42a958fc4c
refactored out http fetching code into Fetcher{}/fetch.go
2024-01-19 18:38:35 +01:00
T.v.Dein
5fa46ff106
Add HTTP retries and the possibility to ignore image download errors ( #33 )
...
added HTTP retry and --ignoreerrors which ignores image download errors, fix #30
2024-01-19 18:38:35 +01:00
T.v.Dein
cca3211023
Enhancement/http ( #32 )
...
* added HTTP debug logging using `-d` or `DEBUGHTTP=1` (headers only)
2024-01-19 18:38:35 +01:00
T.v.Dein
dce7604afb
fix #30 : revert default adnamedir to just use the slug as before ( #31 )
2024-01-19 18:38:35 +01:00
0fd9b519d1
fixed changes on kleinanzeigen.de:
...
- Meta did not contain condition and category together anymore, they
removed the category. Therefore fetching (that is, validation)
failed.
- Now we extract the condition and category directly.
- On top, category now includes the whole category tree.
- unit tests had to be tweaked for this measure.
2024-01-19 18:38:35 +01:00
6b7f727449
fixed utf8
2024-01-19 18:38:35 +01:00
5abbab9527
added template for ad directory, by default include id now
2024-01-19 18:38:35 +01:00
T.v.Dein
e03c7debb6
remove duplicate license badge ( #28 )
...
* remove duplicate license badge
* fix badges
2024-01-19 18:38:35 +01:00
1d2483d18f
portable error check
2024-01-19 18:38:35 +01:00
b17f4f0f3e
also added coverage report+badge
2024-01-19 18:38:35 +01:00
4a91167871
put all tests into main_test.go, more failure mode tests and verify
2024-01-19 18:38:35 +01:00
0baaf6f38b
better error message on 404
2024-01-19 18:38:35 +01:00
42182bb6c9
add commandline main() test units
2024-01-19 18:38:35 +01:00
8455c193eb
pass a io.Writer to loggers and outputs so we can test the cmdline
2024-01-19 18:38:35 +01:00
d1faa10a52
added more invalid tests
2024-01-19 18:38:35 +01:00
e28137bf9b
upd httpmock+deps
2024-01-19 18:38:35 +01:00
1ff5c240c8
put ad code into separate file, enhance error checking
2024-01-19 18:38:35 +01:00
T.v.Dein
f893f9c3d7
Test/add mock tests ( #24 )
...
* add scrape unit test using httpmock lib
2024-01-19 18:38:35 +01:00
T.v.Dein
c4e88d98f2
fix linter errors ( #23 )
2024-01-19 18:38:35 +01:00
T.v.Dein
0cca387982
add ci pipeline ( #22 )
...
Co-authored-by: Thomas von Dein <tom@izb.net >
2024-01-19 18:38:35 +01:00
T.v.Dein
9e619fb3c5
Doc/add prior art ( #21 )
...
* add mor prior art
2024-01-19 18:38:35 +01:00
T.v.Dein
0fdfed2929
added windows screenshots ( #20 )
...
Co-authored-by: Thomas von Dein <tom@vondein.org >
2024-01-19 18:38:35 +01:00
