[Helpers] K-list archived digests & the FST links...

[Mystress Angelique Serpent]

At 06:32 AM 4/25/2005, Rich wrote:
> >     I tried for years to find someone, some way to put up the K-list
> > archives as a few big database text files +  a program that would search
> > the database and spit out an html page containing many emails with that
> > keyword, or an author and date range. Even my druid took a whack at it.
>
>Okay, in summary, I think google has it handled as far as searching goes,
>why not add a google search box directly into the site (this can be found on
>many other sites).

  1.) We already have a google box, it does not work because it conflicts 
with the other search engine. It probably just needs to be moved to its own 
page.

   2.) Google has not indexed all the archives. Despite being invited back, 
recently. In fact, I think there were many thousand more, before I invited 
the googlebot to revisit.

   Google the domain, you get:
Results 1 - 50 of about 841 from kundalini-gateway.org


> >    In the end I had to give up, it seemed the only way to get the archives
> > back online was to convert it to tens of thousands of individual html
> > files.
>
>Why not do it this way... Have database files for each year's messages and
>generate the page automatically from the database. I can do this for you.

    Yes, that is what I always wanted.
   Just one thing... will it still be easy to remove names when people 
request it? Actually, this would probably be less of an issue if the 
archives are no longer searchable by google...

>This way you only need to manage a few pages instead of 1000's. The linking
>and searching would still be there but the current catalogue under google
>would be lost, unless you leave all the old pages in the site.

   It appears they do not exist anyway... We could leave them up for a 
while. The important thing is to get them more readily searchable.

    Hmm.. I just realized that I won't be able to link FST lessons to 
individual posts if the archives become a database... So, yeah, I guess we 
leave the old ones up!! LOL!!



> >    Fortunately, I found a program which does that automatically, but I
> > still have to tweak it a lot after, spam proof all the email addresses,
> > and
> > convert the humungous years index into monthly index files by hand.
> > Hillary
> > helped with that part, last time.
>
>I'd have the index automatically created out of the subject lines of each
>message.

   Yes, it does... but I do a whole year at once so I get an index of about 
7000 links that has to be split into months, by hand.


> >     I also usually spend some time doing a search and delete of hotmail
> > spam sigs, unsnipped digests, and other useless junk. Much easier since we
> > got off yahoo.
> >    I could skip that part, but I'd rather take the time to do it, than
> > have
> > my site spamming for hotmail, google, msn, etc..
>
>I guess a lot could be taken out by search and replace but in my experience
>it's not as simple as this, due to line breaks and changing text in the
>message footers etc.

   Yup, it takes hours...

>The effective way would be to go through each message
>but it's darn time consuming as sometimes people quote another's message and
>include the footer as well so have to check for quoted signature lines too!

    I usually do a combination of both, searching for an individual word, 
like hotmail or MSN and opening all the files that contain it, then search 
and replace on variations, getting the last of them by hand.  Homesite can 
have hundreds, perhaps thousands of files open at once, and it doesn't 
automatically save changed files when you search and replace on all open 
documents. The title tab of the changed, unsaved files appears blue... so I 
just keep running searches till all titles are blue!
   Sometimes I'll also pull up all the posts from someone I know has a long 
spam sig or doesn't snip... I'll also pull up all the longest files, they 
are usually unsnipped digests.

> >    on a related note... Richard has finally solved part of an ongoing
> > problem with the years of digests, and converted them all back to
> > individual emails. Yay Richard!!
>
>It was a labour of love. My search for understanding drove me to do it
>propelled by Goddess, to create an easily searchable (pinpointing words in
>messages) and thread viewable format of every message that I could use in my
>MS Outlook as well as other Unix style readers (Firefox/Eudora/NetScape).
>
>I got the golden carrot nearly as I had finished the task - learning
>something so relevant to the concern I had that I burst out laughing in the
>night when I couldn't sleep. I think the original author wrote it about 5-6
>years back.

   Yeah, there is a lot of good stuff in the early years archives.... we 
were the only K-list and we had quite a few different experts pass 
through.  That is why it sucks that they are only digests... however... 
There may be individual posts that are not in the archives. I'm sending you 
a CD of all my K-list mbx files. Made it for you last week... :)