Posted originally on The Conservative Tree House on November 24, 2020 by sundance
First things first… Again, my most sincere appreciation to all those voices who have called, written and contacted us expressing support. The level of support during this effort to silence our voice is overwhelming and very much appreciated.
The people at WordPress open source software are apologetic, and they point to the people at WordPress.com/Automattic (the business) as the issue; two different groups.
On the technical side of the equation, there is a team of independent website engineers, platform designers and exceptionally skilled technology folks working to assist us. They are working brilliantly on this process, and our CTH community would be lost without their diligent efforts.
We are blessed to have brilliant people on task, focused and executing. To say this is a considerable task would be an understatement. It is very time consuming.
At first blush, the migration seems a simple process of exporting and importing the current CTH data…. until you realize the sheer scale of data involved. There are approximately 35,000 research articles and more than 7,200,000 comments in our decade-long archives each with an independent url to handle. The first step to secure the research library resulted in an export base of 60 files, each file holding massive amounts of data for transfer.
Ten years of graphics, pdf’s, charts, images, videos and millions of internal citations within that library equates to a lot of data. Combine that with a customary 10,000 to 20,000 word daily output of analysis, and then thousands of daily unique comments, and the server space needed for file transfer (as well as the import time) was a seriously unknown variable.
So, the team built a temporary site to see how long the transfer would take. After initial efforts, the engineering realized they could not auto-run the import. Simply the issue was overloading server capacity and few systems could handle the load.
The result is the 60 files can only be loaded in batches of five at a time. Each file takes approximately 30 to 45 minutes to import. The test run took place over two+ days, and we identified a total import time of 30 to 60 hours.
By taking this painstaking approach, we are ensuring that all internal links do not result in a “404” dead link notification. Meaning the library will remain intact, and all external links that lead to the CTH library will be able to find them. Additionally, and/all citations within the CTH library will be retained, and people will be able to use all the embed links; and also importantly, all prior links to your previous comments are retained.
After we knew the architecture and data import would take approximately two days; and adding the anticipated scale of bandwidth to foresee; we have signed up for host data services with an organization that should be able to handle the volume. We are soon to transfer the data from the temporary site into the permanent site on permanent servers.
NOTE: During the transition and data transfer we will need to shut down commenting function for CTH (WordPress) until CTH 2.0 launches. The mirroring of the site during this process will take (based on the test run) approximately 40 hours +/-. As a result, it is highly likely comments will be closed for around two days. Unfortunately, there is no other way to pull this off; it is simply a matter of volume and scale.
We will continue to post new articles and you will be able to see them, you just might not be able to comment on them until the full data transfer to Treehouse 2.0 is complete. Likely, I will have to cross-post to both websites during this transitional phase.
[Remember, the articles from CTH/CTH 2.0 cross-post onto Twitter, and you can also comment there. Or use the Twitter link to return to the CTH 2.0 article when live]
On the good news side… the functionality of the new site should be similar to what you are currently experiencing. There will likely be quirks after the launch that will need to be worked out, but the user experience will hopefully remain (essentially) as it is now, albeit you will not need a WordPress account for functionality. We are using the WordPress open-source software, so most of the current site functions should transfer.
There are some proprietary WordPress.Com features that will not transfer; however, we will immediately begin a priority process of building-in those features via plug-ins where possible…. there are a variety of add-ons that can enhance the new site and even make it better. We will use those features when feasible, and when possible in our budget.
♦ Some exceptionally kind people have requested a subscription platform; however, it has always been my mission to provide the raw and cited details, with an opinion toward what it means, totally free for everyone to use as they choose.
I do not consider my finished products proprietary, because they can always be made better by smart researchers who can find even more details for a bigger story, and make it even better. The site openness and accessibility is as important to me now as it was when The Last Refuge of misfits first assembled 10 years ago. CTH will never charge for access to the truth as we can discover it.
Just as many generous people had excellent suggestions for website hosting; however, many of those were disqualified simply because they cannot handle the volume of CTH traffic without major upgrades and CTH would be paying for those upgrades. The results of the test site disqualified a lot of hosting options. We simply have a ton of data and a great deal of bandwidth use on a daily basis. Our selected host has the capacity (fingers crossed) to handle the scale we need.
♦ The domain name will remain the same: “TheConservativeTreehouse.com”, and should be seamless in transition to the new website. However, after CTH 2.0 launch you may need to clear your browser “cookies” and “cache” so that your search or bookmark reaches the new servers accurately.
How you can tell – If you find yourself only finding older articles when you visit CTH this will be the problem. Clear your “cache and cookies”, search or use your bookmarks for CTH again, and your browser should redirect you to the correct website.
We are up against a deadline of December 2nd – As a result, the graphic design of CTH 2.0 should not look much different. The site will (should) appear the same to users until we go through a process of modifying the design but retaining the theme. Our website theme is over a decade old; we are keeping it but will be refreshing some of the aspects (like the blogroll etc). On an independent site we will have options to add enhancements and make the site carry more features. That part should be fun.
When the comments go live again, please be patient if you get hung up in a moderation cue. One of our challenges will be the absence of Askimet spam filtering. It may take time to release some comments if they are hung up in moderation. The site admins do a great job with comment moderation and release so it shouldn’t be too much of an issue; and quite frankly, there are likely to be benefits from not having the WordPress software filtering through the comments as they post. Spam removal will be a priority.
Please bear with us as we work through some of the challenges of comment moderation in our new platform. It should be seamless, but just in case… well, patience is appreciated. Be assured we will work diligently during this go live phase to deal with any comment issues as rapidly as possible.
♦ Random Stuff – You may not notice, but your user experience is a top priority to us. One of the ways we do this is through opening links in a new window on your browser. As you click on any citation in a CTH article that action opens a new window in your browser; and when you “x” out of that window you return to your original place in the article. This was always a standard convenience approach a decade+ ago on websites.
However, as people started monetizing their sites they realized if they forced the person to exit and come back they could inflate their clicks or impressions. 80 percent or more of websites today push you out of the article when you click a citation and force you to return to finish reading. CTH never subscribed to this inconvenient approach, and we don’t intend to do it now. Each link will open a new page in your browser. We will not inflate our statistics by forcing you to go back and forth just to read citations or more info.
Everything is essentially ready to begin the site changeover. As soon as I know when that process will initiate, I will let you know. Once activated the ability to comment will be closed for approximately one-to-two days (30-50 hrs), though we hope to make it as quick as possible. As soon as the changeover is complete, the commenting function will return to normal…. and you will be on CTH 2.0
Any questions feel free to ask below.
Again, thank you all for your support, this is your community. All of the Treehouse admins are stewards for this place, this Last Refuge, where you are what matters.
The key to Treehouse 2.0 is recognizing we are building something within a system that wants to see us removed. Being proactive will hopefully make us better stewards for our conversation. Remember, YOU are the important part.
Without you CTH doesn’t really have a purpose. I could stand outside shouting at trees with no impact; but together, sitting on the figurative porch, the conversation is rich with unique skill-sets, subject matter experts, and life-long experiences that make discussion so much more valuable. Myself and the CTH admin team cherish the value you represent.
Love to all,
~ Sundance
…Never fight until you have to. But when it’s time to fight, you fight like you’re the third monkey on the ramp to Noah’s Ark… and brother, it’s startin’ to rain.
Centinel2012 