[Fred Broca]

I was just on the database and I was wondering why aren't the weapons named? Seriously, it isn't like we don't know the names of the weapons. Hell, when I should actually be working for a living I got the sudden urge to find out all of the names Ninja Force weapons on the JOE Weapons and Their Real Life Counterparts... thread, (https://www.hisstank.com/forum/g-i-j...nterparts.html) and I added them so they should be on that section.

I think it's fair to say that between every member of the broad we have every G.I. JOE from A.R.A.H. down and we should be able to submissions to that section. HISS Tank should be the JOE equivalent to Wikipedia in terms of being able to add content to the site.

[the Scream Man]

GI Joe can't even keep its Wiki up to date, let alone something with more details and work that YoJoe is. Which is a real shame.

[MagicWazard]

I guess someone in charge of the site had some plans for expansion, as they added a Classified section to the "Toys" category sometime last year, but it's never been built out; it just leads to a 404 error. There was also a period of activity early last year where several news- and blog-type articles got posted, but it's been silent ever since.

Hopefully the site will start getting updated again, or at least won't fully go away; heaven knows how often I've consulted their database the last few years and how much help it's been to me.

[Dusty Ayers]

Quote:

Originally Posted by MagicWazard

I guess someone in charge of the site had some plans for expansion, as they added a Classified section to the "Toys" category sometime last year, but it's never been built out; it just leads to a 404 error. There was also a period of activity early last year where several news- and blog-type articles got posted, but it's been silent ever since.

Hopefully the site will start getting updated again, or at least won't fully go away; heaven knows how often I've consulted their database the last few years and how much help it's been to me.

Couldn?t agree more. I?m on the site a couple of times a week at least. Great reference site.

[THEDARKONE508]

Quote:

Originally Posted by sisco

I downloaded an offline archive of the entire Yo Joe site using HTTRACK. Its a freeware website archiving software that lets you make offline versions of websites. Link to it here: https://www.httrack.com/

Even though I have the Ultimate Guide to GI Joe book I still use Yo Joe. Its an invaluable archive that I want to have a copy of in case the site ever goes down.

Quote:

Originally Posted by jabalong

Interesting. I'm curious, how a big an archive is the YoJoe site?

im interested as well. im starting to fill up my second hard drive with 3d models and rpg pdf's.

is there a way to take out info that's not needed? like i would want to omit the forum, market, news, and contests.

can you edit your collection info on it but locally?

[sisco]

Size depends on how much you archive. The archiving software has a lot f parameters you can set and how deep it goes and how many links it follows. The software is similar to the Internet Archives so theoretically you could end up downloading every site Yo Joe has a link to and then all the Sites those sites link to and enventually fill up a drive.

If you set the parameters right 5-20 GBs in size depending on which part of the site you want. The archiver grabs every thing on the html and makes copies of a lot of redundant things so it adds up.

I suggest finding a small website with not much on it and doing a practice archive to get the archiver settings right for what you want and to mess around with the program before attempting to archive a big site like Yo Joe.

For example I told the archiver to ignore the Yo Joe forum urls, the marketplace, ebay, facebook, google, and to not leave the Yo Joe domain, added a lot of Yo Joe's advertisers urls to the exclusion list as well. I entered the Action Figure section, Vehicle section and comics sections urls as the Urls to archive instead of Yo Joe.com. This kept the archive limited to the toy/comics instead of Yo Joe front page news.

The archival process is also slow, do it on a pc you can let run for a full 24 hours at least. It automatically throttles its speed so it doesn't overwhelm a sites bandwidth, it usually runs downloading at under 100kbs on most sites even if you tell the program not to use a speed limit or it will follow a sites Robots.txt instructions for archiving that sites like the Internet Archive and google cache use. The program runs in the background, takes up very little pc resources, and very little of your networks bandwidth so you can leave it running while gameing, sufing the net, watching videos, etc... Its a set it up and check on it every few hours type of thing.

You could probably get everything you want in less space by manually saving each page from your browser, which takes a lot of time and when you want to view it you will have to open each page individually from its folder and Hyperlinks most likely will not work.

The benefit of an archiving program is that its done automatically and makes a full navigable website on your hardrive that works just as the original website, the hyperlinks will take you to the webpage or imagefile on your hardrive instead of to the weblink. And you can copy and paste the archives master folder and move it to other pcs or hardrives and it will still work the same.

Final note the archive should stop automatically but if it runs over 36 hours or the urls it starts showing as downloading are way off the websites domain, I'd hit cancel on it. When you hit cancel it will quickly make a note of all the domains it had left to scan without downloading them and make a working archive of the site you can then browse on your drive. Past 36 hours it most likely got everything you wanted of the site on your hardrive and was just heading down the internet rabbit hole of hyperlinks. When you open the My Websites folder on the hardrive it will have labled folders of each domain it archives a lot of these folder will be ads or partner sites that the bot followed or got sidetracked by, you can safely delete any folders that are not the Website names folder or an image hosting service if the site uses one for its images.

[THEDARKONE508]

Quote:

Originally Posted by sisco

For example I told the archiver to ignore the Yo Joe forum urls, the marketplace, ebay, facebook, google, and to not leave the Yo Joe domain, added a lot of Yo Joe's advertisers urls to the exclusion list as well. I entered the Action Figure section, Vehicle section and comics sections urls as the Urls to archive instead of Yo Joe.com. This kept the archive limited to the toy/comics instead of Yo Joe front page news.

man, you arent kidding that this archives everything. is it pretty intuitive?

[sisco]

Quote:

Originally Posted by THEDARKONE508

man, you arent kidding that this archives everything. is it pretty intuitive?

Yeah its got an easy to understand UI and when you hover over things it gives a description of what the option does.

Basically you start it up, hit next, name your project like "Yo Joe" hit next.

Select Download websites in Action drop down (its the default), change it to Continue Interrupted Download if it got interupted, or change it to Update existing if you want to get pages added with a site update.

Copy and paste the Website URLs you want.

Choose Set Options. Uncheck use Proxy for FTP if you don't use one.

Choose Scan Rules, Check the box for gif, jpg, png, tif so it gets images. Feel free to check the other file boxes if you want. Click Exclude Links and add anything you want to exclude. Like to avoid getting facebook links choose Links Containing in Criterion and post facebooks url in the string box. Do this for anything links you know you don't want. Its ok if you leave it blank it will just take longer to download everything. The program includes some ad servers already blocked by default.

Under Limits remove or set any you want, you can limit the programs speed if you have really bad internet and still want to surf the web. The main thing I usually do is set Max External depth to 1 so the bot doesn't stray far from the site and will usually still grab images hosted off the web sites server. Feel free to set or remove any limits you want.


Under Links section you may want to check the Get Non-HTML files related to a link, eg external ZIP or Pictures depending on the website. If You are archiving a primarily Text based site like say all the GI Joe category on Fanfiction.net. I would suggest opening the Links tab and selecting GET HTML files First! So it will focus on downloading the stories before images.

Rest of the stuff is usually best left on default.

One more thing. When you exclude stuff most times it will still create a folder with a 0byte file with the excluded stuffs name. Like if you block an ad it will create a 0 byte file with that ads name. This is so the local archived Websites can display the site as it looks online but instead of showing an ad that was excluded the local website will load the 0byte file as a blank space on the page.

Theres a lot of neat options you can really get into and customize the heck out of whats downloaded, how its displayed, archived etc. Its a really robust program for archiving.

Edit: Oh the Website will be saved in whatever folder you choose,C:\My Websites by default. Open that folder and click Index.html to load a webpage listing all the sitrs you archived. Click the project name you named the site on the index and you will be accessing the site offline using all the files it archived, browse the site and click links, images, etc like you would on an online site. If you hit something you didn't archive it will show a 404 error or something similar, just hit back to go back to the previous page.

[Ruin]

Quote:

Originally Posted by AWOL

Imagine if Hasbro made their own database.

It would require a paid subscription.

[AWOL]

Quote:

Originally Posted by Ruin

It would require a paid subscription.

Someone else mentioned this earlier. It's probably true.