Overstock (and now price) history charts

Discussion in 'Talk' started by Cheese Wheel, Sep 28, 2012.

  1. Cheese Wheel That guy who makes charts

    I've started collecting overstock history data, and you can query this for a chart covering the last 24-hour period for any item. There's currently no front-end you can use to select the item you're interested in (I may be adding this) but you can enter the item and quality IDs if you know them. For example, for keys:

    http://www.chrishowie.com/tf2wh-charts/index.py/?item_id=5021&quality_id=6

    Data has 1-minute granularity. If rendering the chart is really slow... get a faster computer. :p

    Update: New overstock/price history chart website is at http://tf2wh-charts.chrishowie.com/ -- the URL above will simply redirect here.
  2. straddler Member

    Thats really cool.

    But.

    What your army of fans really wants is for you to give that treatment to price changes, with more than 24hr historical data. Nudge nudge, wink wink :D
    tardlife likes this.
  3. Cheese Wheel That guy who makes charts

    That may or may not already be in the works.

    Note that the overstock data set is not small. The history table is growing by about 114 MB per day (including indexes). I'm going to have to start rolling week-old data off into archived cold storage at some point.

    Pricing data, since it changes far less frequently, should be able to be retained hot for longer periods of time.
    tardlife likes this.
  4. straddler Member

    Is it necessary on the overstock to have a data point every minute?
    tardlife likes this.
  5. takepwn New Member

    Wow, that was a HUGE jump for the key stock just now.
    tardlife likes this.
  6. BenWah New Member

    Yeah they added 1000 keys at once looks like.

    I think the people working the levers don't want folks selling keys or refined metal any time soon!

    My hopes of getting more credits just went up in smoke.
    tardlife likes this.
  7. Cheese Wheel That guy who makes charts

    Sort of. I can omit consecutive equivalent data, but then I don't know if I lost data or not. And right now, the charting solution I'm using doesn't let me specify the width of each data point -- all points on the X axis are spaced equally. So if I didn't have a consistent amount of data points per unit of time, the X axis scaling would be off.

    I may address this at some point, but it's rather low on my priority list at the moment.
  8. Cheese Wheel That guy who makes charts

    To give an update on this... I am currently collecting pricing data. The amount of data I have is not yet to the point where charts would really be helpful in any way. Additionally, I'm currently performing a major migration on my httpd configuration, switching not only httpds (Apache -> nginx) but also the entire URI site-mapping scheme. This will take some time to test, and I won't be doing much web development while this is ongoing. By the time I'm done, I should have enough historical pricing data that charts will become useful.

    tl;dr version: I'm doing a bunch of shit, not enough data yet anyway, stay tuned.

    (I will be working hard to ensure that my existing web services -- including the overstock charts -- are not interrupted during this migration.)
  9. Cheese Wheel That guy who makes charts

    Based on the overstock history data, it appears that this data is updated by TF2WH every 10 minutes. I have therefore altered collection of historical overstock data to occur once every 5 minutes instead of every minute. This should result in faster chart load times, both due to the decrease in the amount of data downloaded to your browser, but because the database won't have to dig up as many pages to fulfill data requests.
  10. Cheese Wheel That guy who makes charts

    All done. http://tf2wh-charts.chrishowie.com/

    Overstock charts go back a day, price charts go back a month. (I don't have a month's worth of pricing data yet; right now you only get about one and a half weeks, but this will go up to a month as time passes.)

    Reply to this thread with any bugs/anomalies.
  11. Jimmy [ScrapBank] Active Member

    Interesting...but all graph hare...weird...50px height
  12. Cheese Wheel That guy who makes charts

    Which browser? The graphs should fill your browser window, whatever size it is.
  13. Jimmy [ScrapBank] Active Member

  14. Cheese Wheel That guy who makes charts

    Ok, I will test with that browser when I get home. I developed mostly on Chrome; if you having that laying around it might be worth a shot.
  15. gambler Active Member

    Refined stock is at 2970 for almost 20 minutes already, it'll go over 9000 when updated
  16. Barrakketh New Member

    Looks like it hit 4562 after the update.
  17. chewi105 New Member

    this site is becoming unusable for anyone trying to sell stuff.
    Menthol likes this.
  18. triddin Administrator

    Oh really? People are actually selling stuff though? So it's obviously usable.
  19. chewi105 New Member

    not the bulk stuff. theyre selling stuff thats understocked by one item. most of the time, its pure chance that someone sells a random weapon, or just a hat that no one sells because its easier just to get a ref for it.
    Menthol likes this.
  20. gambler Active Member

    Supply & demand, and the demand of most items is very low.
  21. Cheese Wheel That guy who makes charts

    If you guys want to talk about whether the WH is usable or not, take it to another thread. I want to keep this thread for bug reports regarding my charts.
    tERROR and Menthol like this.
  22. Cheese Wheel That guy who makes charts

    Fixed on Firefox. (You will have to CTRL+F5 to reload the stylesheet.)
  23. Jimmy [ScrapBank] Active Member

    Nice thanks.
    You can combine price and stock level in one graph ?
  24. Purplecat Member

    I noticed that the overstock graph for B.M.O.C doesn't seem to work. Is that due to missing data?

    I recently messed with parsing the CSV file with a Java program I made under a half hour. It was easy enough I'm surprised I hadn't seen more applications of the CSV of the overstock page (well I backpack.tf does use it to grab warehouse prices I suspect).
  25. Cheese Wheel That guy who makes charts

    I'm not sure how that would work... it probably wouldn't be very readable anyway. I might be able to put both charts on one page though.
    When an item has stock level of 0, sometimes the JSON overstock dump from the WH doesn't include the item at all. Since it hasn't been in stock in the last few days, there is simply no data for the last day. (The last record was at 2012-10-09 16:50:02 UTC.)

    I'll try to add an error message for this particular case.
  26. Cheese Wheel That guy who makes charts

    I'll probably start working on this soon, but note that the data won't line up perfectly. The granularity of overstock data is much higher (since it changes more frequently). The amount of history shown is also different for the two graphs.

    To put things in perspective, for an entire day's worth of history (which is the timeframe used for the overstock charts) there are 17,280 overstock data points, and 4 price data points.
  27. RedMage L33D Moderator

  28. Cheese Wheel That guy who makes charts

    This is because the WH gives me item IDs for the overstock data, but not for the pricing data. I have to manually match up the pricing data with items by searching on the item name, and the stock weapons have duplicates in the schema. The pricing data is therefore associated with whichever stock weapon definition the database finds first.

    This is not as much a bug with the charts site as it is an unfortunate consequence of (a) having duplicate items in the schema, and (b) the WH not providing a price dump with item IDs. If I am able to get a JSON dump of the pricing data (poke poke, @triddin) then I might be inclined to correct the historical data, but right now it is too much effort for too little benefit.

    (Even the overstock data doesn't match up completely -- for example, the Strange Rocket Launcher in the overstock data dump has item ID 18, while Strange Rocket Launchers in backpacks have item ID 205.)
  29. Truckilles Member

    I'm not understanding how medi gun gets a price or stock amount since its not tradeable unless I'm missing something very obvious...
  30. Wallachia Medic of Steel

    You can turn a stock weapon into a tradable item by renaming it then cleaning it to the original name.
    Truckilles likes this.
  31. Purplecat Member

    You don't even need to clean it. Infact if you're bored you can always put in renamed items for fun onto TF2Wh.

    Which probably wind up being turned into scrap by users but oh well. :p
  32. Jimmy [ScrapBank] Active Member

    Is not working anymore ? :(
    From time to time i check some chart but now look like is not updating anymore
  33. CaioXG002 Well-Known Member

    DAT BUMP!
    Seems to be working for me... I guess...
  34. Cheese Wheel That guy who makes charts

    It is updating. What makes you think it is not?
  35. Jimmy [ScrapBank] Active Member

  36. CaioXG002 Well-Known Member

    It shows 2013-01-06 for me... Reset your computer.
  37. Jimmy [ScrapBank] Active Member

  38. CaioXG002 Well-Known Member

    Calm down, man. I made the same mistake as you by not realizing it's the wrong link.
    Confirmed bug on the site tough. There's even an empty part on the left and on the right of the chart.
  39. Cheese Wheel That guy who makes charts

    I've forced a manual update for now. The updating script has been running once every six hours, but it seems to have failed to download the price list since 12/27. I've turned on email monitoring of this script, so if it is failing I should learn why the next time it tries to update.
  40. bob742omb New Member

    An interesting trend I've noticed is that prices for some items have jumped since late December.

    Anyway, a useful tool indeed!
  41. Wolfbane Active Member

    Why are achievement only items in there?
    I saw the "Lugermorph" then below it "Vintage Lugermorph". I'm fairly certain there was no glitch to make tradeable Lugers.
  42. PsychoTink Moderator

    I bet you there was.
  43. RedMage L33D Moderator

    The warehouse has a price for it for some reason, so it gets graphed along with everything else.

    While I don't see any tradeable unique lugers anywhere, it was certainly possible with the giftwrap glitch. It would have to have a "gifted by" text though.
  44. Wolfbane Active Member

    TO THE ALAMO!
    [IMG]
  45. CaioXG002 Well-Known Member

    /\ This image... The fuck is it?
  46. Wallachia Medic of Steel

    It is the Demopan.
  47. Wolfbane Active Member

    Do you know The Demopan, The Demopan, The Demopan? You don't? GTFO FAT MAN!
  48. Wolfbane Active Member

    Oh shit, we're derailing. Sorry Sera!
  49. CaioXG002 Well-Known Member

    I know the Demopan, but that's the most strange usage of it.
    And the topic did just get a necro-bump, I don't think Sera would kills us just because of a picture.
  50. PsychoTink Moderator

    But I might. The picture was unnecessary, but one thing. The conversation after it was a different matter, but it needs to stop. People have been derailing threads WAY too much lately, and I'm sick of it. Next off topic post and I'm likely to start cutting posts out of here and moving them to trolling. Keep on topic or take it somewhere else.
    One Cool Cat likes this.

Share This Page