Writing my bachelor's thesis, I noticed that I had several problems with my workflow:
- Keeping a journal and analyze one's test results is difficult. Doing so and in the meantime running several other tests, knowing which tests are comparable to which is even more difficult. Especially when one's software is regularly changing to meet new needs.
- Succeeding in getting the same test scores today as I got two months ago is surprisingly hard.
I have been doing some thinking about what is missing, and maybe what I want is a program which allows me to say: "I have a bunch of experiment results right here, and I believe they were retrieved some time in May. My automatically generated log book seems to confirm that such a result was created on May 18. Please recreate the state of the program as it was back then, and rerun the program with the same parameters as I used in that experiment."
Essentially, an experiment log book that also stores the commit IDs of the current state of the experimental framework (and in the case where the scientist doesn't use VCS, one could automatically be set up for them). This would potentially solve the second problem, and be the ground work for the solving the first problem, which perhaps could be summarized as "Evernote for the scientist".
My question to you is mainly the one asked in the title, but also whether you have had the same problem yourself, and how you came to solve it. Would you see the value in using something like what I just described?
[link][4 comments]
