We’re still trying to figure out why our Google traffic is off by about 50% in the last two months. Algorythm shift? Google bug? Sun spots? And while I’ve been digging around I’ve found some seriously weird shit in our Google site index. For example, here’s a screen grab of page 25 of their index. Check out the text that Google has supposedly harvested from my wife’s blog catagory [Peggy C, The Other Half » geeky stuff]:

Yesterday it was a bunch of my blog posts that had the same sort of erroneous porny-spammy text scrapes in Google’s index of our site. Today they’re (as far as I’ve looked) gone, but now Google snap-shot of my wife’s blog is poisoned. Where’s Google getting this stuff from? It’s certainly not anything I wrote, or Peggy wrote?

The scrapes on bunch of other posts on my blog simply have the menu bar alt-text instead of the actual post content. I’m sure the page-rank for those posts has fallen through the floor. Oh well.

Normally December is our biggest month of the year, but this December it’s just average. Fortunately since our last Google debacle we’ve developed other avenues of getting our work to market, so we can absorb the hit. But no doubt that are other web-based merchants out there for whom this isn’t going to be a very merry Christmas.

Well something is definately up.

On November 7th, the day after I posted Real Sex, Nina Hartley, and the Googlebot, our Google-driven traffic dropped by 40%, and has stayed there for the last week.

Okay, fine. No one, not even Comstock Films is entitled to high Google rankings.

But while our overall Google traffic has dropped by 40% in the last week, our visitors on the search [nina hartley] have doubled, and that’s on top of the ~400% increase reported in my first post. In fact, [nina hartley] is now our #2 search term, outranking [real sex] or even [tony comstock]. (Meanwhile, our visitors on the search [real sex] have virtually dried up altogether.)

I’m not happy about losing our [real sex] visitors; they were among our best search related customers (second best actually.) But I’m even less happy seeing that once again, Google seems to be dicking around with their sex-related search just ahead of the holiday season, and in there dicking around, they seem to have come up with a picture of our site that just isn’t accurate. We’re getting visitors we shouldn’t get, and people who are probably looking for exactly what we offer are getting sent somewhere else.

I’m a filmmaker, but I am also a merchant, and like any other merchant, this is an important time of year for us. In the last year we’ve done a lot to be less dependent on search-driven sales to make ends meet, but those sales are still an important part of how we’re able to pay our bills. A 40% drop in Google-driven visitors can’t be good for us, and could be very bad.

Google’s search algorithms would seem to have undergone another fairly major revision, at least around sexuality. What makes me say that? In the last month, people finding their way to ComstockFilms.com on the search [real sex] have fallen by about 80%, while people finding their way to ComstockFilms.com on the search [nina hartley] have risen by about ten-fold. Yesterday we actually got more visitors looking for Nina Hartley than we are looking for real sex, which probably isn’t good for Nina, Comstock Films, web-searchers, or Google for that matter.

I suspect this has something to do with the Googlebot trying to distinguish between spamblogs and original content, but who can know for sure?

The good news is that this hasn’t had nearly the same effect on us that the Great Google Disaster of 2006 did. Since last year we’ve taken steps to make sure our fortunes aren’t so dependent on the visisitude of inscrutable Googlebot.

“Google Strokes the Porn Guy”, that’s the headline for the article that ran in the Easter Sunday edition of The New York Post; several hours of genuinely interesting conversation with writer Damon Brown about sex and technology and art and filmmaking, boiled down to by the Post’s editors to 300 words that quite nearly get the story Google story straight. A couple of corrections:

1) I didn’t write to Google. I wrote Violet Blue, who made this post to her blog. Her post was picked up by Boing Boing. Somewhere in that chain of events is where our problem came to the attention of Google’s Matt Cutts, who was responsive and helpful.

2) The Post article makes it sound like I passed on private e-mail. I didn’t. I did point the writer to this post, Google’s Matt Cutts Wants to Know More About Sex, where, with Matt Cutt’s permission, I published our e-mail exchange.

The list of “porn guy strokers” (ugh) should also include Seth Finkelstein, Phillip Lenssen, Barry Schwartz and Danny Sullivan, SEO/Google experts who were extremely generous in helping me getting our Google situation sorted out. Shit like this is nerve-wracking, their moral support was as appreciated as the technical insight!

We’ve had our share of problems with Google and Adwords and the erotophobic bias of the English language. Now it’s someone else’s turn to go through the rinse cycle.

Don’t Be Evil (Or A Dyke, Or Trans), Violet Blue’s Open Source Sex

Of course you can just do what Elora’s Cave does and make sure not to use any of the Google-verborten words that are the meat and potatos of your books anywhere on your website.

It can be hard not to be ashamed when nearly everywhere you turn, if you want to reap the benefits of particpating in the larger culture, you’re told you have to act like you are ashamed; substituting coded language and knowing looks for real ideas and authentic emotions.

(I don’t know what our favorite Hells Kitchen lumber yard is going to do if they ever feel they need to do some Google Ad words.)

I’ve never seen something like this for any other Gooogle search:

Have you?

Scroll down to out the ads Google generated for Violet Blue’s latest Chronical column:

Some things are so beautiful they shimmer!

Well maybe it’s me.

After a week of tearing my hair out and exchanging a lot of email with SEO specialist Danny Sullivan, Seth Finkelstein, and Phillip Lenssen, and coming up with a lot of theories why ComstockFilms.com fell of the face of the earth on some of our most important search terms, this morning, while chewing over it some more I remembered three different, vaguely connected things and maybe, just maybe I found a reason that this has all happened.

Last week, against the worry that some of our feeds might have been looking like duplicate content, we made changes to our robots.txt file. We checked our robots.txt file in using Google’s webmaster tools, and everything seemed fine. It looked like it was doing what it was supposed to be doing.

But this morning when I tried to feed our robots.txt file to Google’s URL removal form, it spat it back out and accused us of bad syntax.

Really? I went back and checked it with Google’s webmaster tools. All a-okay.

But just for fun, I re-wrote the (alledgedly) offending lines, and bingo, the URL remover ate the file like orange sherbert.

A few hours later there are still some goofy things on the first page of our Google site index, but at least our index.html is there too.

My theory is the bad syntax opened a door into all those files we didn’t want the Googlebot messing around in, and that’s how all those nasty, outdated URLs ended up in our site index, and pushed out all our nice new Relevant URLs. Hopefully by tomorrow morning it will look better, not worse, and maybe by the end of next week things will be back to where they were at the begining of last week.

Or not. Time will tell.

In the mean while, we make a lot of progress on ASHLEY & KISHA and I cold-called like a mad man and picked up a few new retailers. I remember reading about the North Korean human wave tactics and how they would simply flow around restance points and envelop the enemy. This week Peggy and I were our own little human wave. The Google issue resisted everything we could come up with, so we flowed around it and claimed victories in production and sales. I’m exhausted, but proud of us. Whether or not this robots.txt thing is the fix, tomorrow morning’s lox and bagels will feel well-earned!

Yesterday was a blood-bath. The Google Site Index for Comstock Films continues to look extremely weird, full of robot.txt excluded pages, and even pages that haven’t been live on our website for years, and our Google search related traffic is in the toilet.

Why does Googles’s index of our site have pages that we took down two years ago? I don’t know. Why does it have pages from directories we’ve excluded from indexing in our robot.txt? I don’t know. (And yes, we checked them using Google’s webmaster tools.) Why doesn’t the site index have our important and well-linked to pages like index.html and main.html? I don’t know. I wish I did, and I wish I knew what to do to fix it.

Adding to my puzzlement, while here the US, ComstockFilms.com is somewhere around page 10 for the search ‘real sex’ our site continues to enjoy a relatively high position for the search in Australia (currently page 1) and the UK (currently page 3).

There’s a certain irony to this. Both Australia and the UK have government-imposed ratings for films, and our DVDs are illegal to sell in most of Australia and in all of the UK.

Coincidentally, a copy of our first film MARIE AND JACK: A HARDCORE LOVE STORY is currently at the Motion Picture Association of America, and we expect a rating decision by the end of the month. No doubt the film will receive an NC-17, which means adults only.

50 years ago there were only two ratings; adults only and general audience. Children couldn’t get into adults only films, not even with their parents. Adult films were for adults in the same way that bars are for adults.

But somewhere along the way we lost the idea of exclusively adult cinema. Yes, the MPAA has an adults only rating, but it’s an economic dead-zone. SHORTBUS went out unrated, rather than bear the stigma of the NC-17 rating. (Many media outlets will not accept advertisements for films rated NC-17, and some theaters will not show them.)

Conversely, the ultra-violent SAVING PRIVATE RYAN, a film that is utter inappropriately for children was given an R by the MPAA, which means kindergardeners can go see it, provided they are accompanied by a parent or guardian.

I can understand (both commercially and ethically) the MPAA wanting save PRIVATE RYAN from the burden of carrying the NC-17 rating, but the idea that it’s a film suitable for children of any age, provided they are accompanied by an adult is manifestly absurd. Although what can be shown is far more permissive than it was 50 years ago, in the process we’ve given up the space for adults to experience genuinely adult films.

Meanwhile, in Australia and the UK, there is a litigate adults only rating. In Australia and the UK, rated-R means no one under 18, and that’s the rating films like SAVING PRIVATE RYAN receive in those countries, (and SHORTBUS and DESTRICTED because they’re not intended to arouse!)

(Lest I sound too in love with the Australian and UK system, please remember that their systems are manditory and governement imposed, and that’s why our films are illegal to sell in Australia and the UK.)

MARIE & JACK will have its MPAA rating by the end of the month, an NC-17; and have no intention of running from the rating. In fact, we intend to embrace it, to wear our NC-17 as a badge of honor.

Like the rest of our work, MARIE & JACK is a film for adults about the very adult experience of sex. No, it’s not okay to bring your kids. Maybe you think it’s okay to bring them to see SAVING PRIVATE RYAN, but you can’t bring them to see MARIE & JACK. It’s grown-up time now, and this is a movie for grown-ups to watch with other grown-ups. We don’t want to here your 12 year-old giggling, or you stammering when she asks an awkward question. Get a sitter, or stay home, or wait for it to come out on DVD.

A few years ago I went to a very Sad Place to make a movie about some very Bad Things. Along the way I taught myself a Silly Trick.

Not knowing quite what do do with all the anxiety about being in a Sad Place making a movie about Bad Things, I bundled up all my worry and put it in the pit of my stomach.

When I got home from the Bad Place and started editing the movie about the Sad Things, I started to feel sick. The obvious answer was that I must have gotten a Bad Bug in Sad Place. But when no manner of poking or prodding or looking at my shit under a microscope could find any Bad Bugs, my doctor and I were forced to conclude that I was manifesting a classic symptom of stress. I had quite literally worried myself sick.

Knowing my inside were not being devoured by a Bad Bug was a huge relief. I didn’t get better right away, but I did get better. But the Silly Trick stayed. I don’t know where my worry used to live (probably nowhere because I was too stupid to worry,) but now it lives in my belly. Take it from me, this is a stupid place to put your worry.

This morning I’m worried about the fact that some time in the last 36 hours Google’s index of ComstockFilms.com has become seriously weird and our search-related traffic has dropped by 50%.

I’m worried that we may have inadvertently done something to trigger this strange re-indexing and I don’t know how do fix it. But I’m worried that we may have done nothing, and that we are powerless to fix it.

But this mornign I’m also worried about the fact that Ashley & Kisha’s interview is still over an hour long, and I’m already into hard choices about what to keep and what to let go.

If it were Summer, I would take the morning off. I’d go to the beach or the lake, or go fishing, or do something else to take my mind off of things. But it’s not Summer. It’s the dead of Winter, in the middle of a cold snap, and we don’t even have any snow. There’s not much to do, except fret, and worry, and belly ache. Which I am, and it does.

But it’s not all belly aches and belly aching this morning. Here’s a nice passage from Ashley & Kisha’s interview, about how Ashley used a deftly applyied tongue to seduced Kisha, and what it means to be out and in love:

http://www.comstockfilms.com/quicktime/akpreview0207.mov
(approx. 6 minutes, 20megs.)

Still a rough cut, so please, be cut me a little slack on the mix and the edits. I can see how I could cut it down by 30%, but I’m worried about being too heavy-handed. Somes the best thing to do is just give people the time to say their piece.