![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Stats functionality restored
Right, I think I've finally created a working command-line script to highlight, record and allow me to query changes in my AO3 stats on a monthly basis -- functionality which the site itself conspicuously fails to supply :-p
It relies on keeping a local database file which gets updated with lines for every story in the system, even for those which haven't actually changed since the last save, so it's not very space-efficient -- and the script has to scan through the entire database in chronological order to identify the most recent values, so it gets progressively less and less efficient as the database increases in size. But it's written in awk, which is optimised for analysing huge quantities of line-based text data, so given a reasonably fast processor it should remain usable for a fair while.
Theoretically, given awk's ability to process multiple input files, it would be reasonably easy to add functionality to support having old archive databases which only get scanned when you're actually searching for back data, while keeping the 'latest' database file to just the last X thousand records, but since I don't plan to save the data every day -- I don't get enough hits at the moment for the figures to change that rapidly! -- the database shouldn't grow all that fast, even if/when I upload more stories to the site.
The script is based on the old FFstats script I developed in order to highlight which stories had received hits recently (e.g. since I last saved the status), plus the added ability to display the number of hits over the whole of a selected month, information made available natively on FFnet but not provided by AO3. The latter displays only data for the entire lifetime of a story, and it's nice to be able to see how the figures change over time.
So you can display either a list of which stories received hits since your last download/save (with the option to record that data and restart the comparison from zero, or to exit without saving and keep accumulating more hits until the differences are sufficiently numerous that you start losing track of where the changes are), or a list of which stories received hits over a user-specified month. I didn't provide any flexibility to query the figures over any period other than a calendar month, simply because I wasn't used to being able to do that in the old days and hence haven't really felt the need ;-)
I only have a relatively limited amount of data saved so far during the development period (from 19th Jan onwards), so I haven't been able to test exhaustively for all possibilities, although I've experimented with fake figures for testing purposes. More bugs will probably show up with use.
Default operation (just save out stats page, run script and hit RETURN without entering anything at the initial prompt -- the display shows those stories which have received hits since the last save and what period that covers)
Querying back data: enter a month (and optionally a year) to display, and the change over the dates recorded for that month is shown. (Or, in fact, the change since the last saved date *preceding* the start of that month's figures, since unlike FFnet the data doesn't get automatically reset at the start of the month and I'm not saving daily!) In this case no save prompt is given; the database can *only* be updated with a record of what has changed since the last save.
I made the deliberate choice to extend this option forward to cover the latest AO3 data even if this is not yet actually saved to the database, when the current month is selected:
Database changes over a past (and in this case partial) month:
Results for bad input values:
Results after saving file:
It relies on keeping a local database file which gets updated with lines for every story in the system, even for those which haven't actually changed since the last save, so it's not very space-efficient -- and the script has to scan through the entire database in chronological order to identify the most recent values, so it gets progressively less and less efficient as the database increases in size. But it's written in awk, which is optimised for analysing huge quantities of line-based text data, so given a reasonably fast processor it should remain usable for a fair while.
Theoretically, given awk's ability to process multiple input files, it would be reasonably easy to add functionality to support having old archive databases which only get scanned when you're actually searching for back data, while keeping the 'latest' database file to just the last X thousand records, but since I don't plan to save the data every day -- I don't get enough hits at the moment for the figures to change that rapidly! -- the database shouldn't grow all that fast, even if/when I upload more stories to the site.
The script is based on the old FFstats script I developed in order to highlight which stories had received hits recently (e.g. since I last saved the status), plus the added ability to display the number of hits over the whole of a selected month, information made available natively on FFnet but not provided by AO3. The latter displays only data for the entire lifetime of a story, and it's nice to be able to see how the figures change over time.
So you can display either a list of which stories received hits since your last download/save (with the option to record that data and restart the comparison from zero, or to exit without saving and keep accumulating more hits until the differences are sufficiently numerous that you start losing track of where the changes are), or a list of which stories received hits over a user-specified month. I didn't provide any flexibility to query the figures over any period other than a calendar month, simply because I wasn't used to being able to do that in the old days and hence haven't really felt the need ;-)
I only have a relatively limited amount of data saved so far during the development period (from 19th Jan onwards), so I haven't been able to test exhaustively for all possibilities, although I've experimented with fake figures for testing purposes. More bugs will probably show up with use.
Sample output
Default operation (just save out stats page, run script and hit RETURN without entering anything at the initial prompt -- the display shows those stories which have received hits since the last save and what period that covers)
Display data for which month? (1-12)
(RETURN to display changes since last save by default)
From 05/02/2022 to 07/02/2022
A Thing of Value Hits: +1 Kudos: +0 Bookmarks: +0
The Man Who Knew Too Much Hits: +1 Kudos: +0 Bookmarks: +0
Appraisal Hits: +2 Kudos: +0 Bookmarks: +0
A Child of the Law Hits: +1 Kudos: +0 Bookmarks: +0
4 changes
SAVE? (y/n)
*Querying back data: enter a month (and optionally a year) to display, and the change over the dates recorded for that month is shown. (Or, in fact, the change since the last saved date *preceding* the start of that month's figures, since unlike FFnet the data doesn't get automatically reset at the start of the month and I'm not saving daily!) In this case no save prompt is given; the database can *only* be updated with a record of what has changed since the last save.
I made the deliberate choice to extend this option forward to cover the latest AO3 data even if this is not yet actually saved to the database, when the current month is selected:
Display data for which month? (1-12)
(RETURN to display changes since last save by default)
2
In which year?
(RETURN for current year)
From 29/01/2022 to 07/02/2022
The Choices of Raoul de Chagny Hits: +4 Kudos: +0 Bookmarks: +0
A Thing of Value Hits: +3 Kudos: +1 Bookmarks: +0
The Man Who Knew Too Much Hits: +1 Kudos: +0 Bookmarks: +0
Appraisal Hits: +2 Kudos: +0 Bookmarks: +0
A Child of the Law Hits: +1 Kudos: +0 Bookmarks: +0
Shall We Dance? Hits: +1 Kudos: +0 Bookmarks: +0
6 changes
*Database changes over a past (and in this case partial) month:
Display data for which month? (1-12)
(RETURN to display changes since last save by default)
1
In which year?
(RETURN for current year)
From 19/01/2022 to 29/01/2022
Lost and Found Hits: +1 Kudos: +0 Bookmarks: +0
Count Philippe Takes a Hand Hits: +2 Kudos: +0 Bookmarks: +0
The Choices of Raoul de Chagny Hits: +4 Kudos: +0 Bookmarks: +0
The Writing on the Wall Hits: +3 Kudos: +0 Bookmarks: +0
Teach Me to Live Hits: +1 Kudos: +0 Bookmarks: +0
A Thing of Value Hits: +2 Kudos: +1 Bookmarks: +0
The Man Who Knew Too Much Hits: +1 Kudos: +0 Bookmarks: +0
Appraisal Hits: +3 Kudos: +2 Bookmarks: +1
The Sons of Éléonore Hits: +5 Kudos: +0 Bookmarks: +0
Shall We Dance? Hits: +2 Kudos: +0 Bookmarks: +0
10 changes
*Results for bad input values:
Display data for which month? (1-12) (RETURN to display changes since last save by default) must In which year? (RETURN for current year) 2023 No records for 00/2023 *
Results after saving file:
Display data for which month? (1-12)
(RETURN to display changes since last save by default)
From 05/02/2022 to 08/02/2022
A Thing of Value Hits: +1 Kudos: +0 Bookmarks: +0
The Man Who Knew Too Much Hits: +1 Kudos: +0 Bookmarks: +0
Appraisal Hits: +2 Kudos: +0 Bookmarks: +0
A Child of the Law Hits: +1 Kudos: +0 Bookmarks: +0
4 changes
SAVE? (y/n)
y
Updating file...
*
Display data for which month? (1-12)
(RETURN to display changes since last save by default)
From 08/02/2022 to 08/02/2022
0 changes
*