Log in Register FAQ Memberlist Search pcHDTV Forum Index
pcHDTV Forum

pcHDTV Forum Index -> HDTV programming and listings -> New program: psipguide
Post new topic  This topic is locked: you cannot edit posts or make replies. View previous topic :: View next topic 
New program: psipguide
PostPosted: Tue Mar 02, 2004 8:05 pm Reply with quote
dlarrick
 
Joined: 22 Sep 2003
Posts: 58
Location: Outside Boston, MA




I have written some code to extract PSIP guide data from the ATSC stream. Those who have seen my testinfo program or MythTV's hdtvrecorder will find some of this code familiar.

I wrote this because one of my local PBS stations (WGBX in Boston) began sending guide data for their multicast subchannels in PSIP. (Of course, they have stopped doing so, but I believe the longterm plan is to do so again.) This guide data is not available anywhere else from what I can see, so if I want to record their content, I'm gonna have to use PSIP.

You can get the code from http://jekyl.ddts.net/doug/psipguide.C . Note the following caveats:

* I am not processing CRCs in this code. Eventually it's bound for MythTV, which has routines to deal with the CRC, so this will happen at that time. But as a standalone program, it could use CRC.
* My scheme for figuring out if we've got all the data could probably use some work (see the comments at the top for description)
* I have hardcoded the leap-second offset for GPS time; this should be parsed out of the System Time Table instead
* As a standalone program, it would be helpful to spit out valid XMLTV-format data rather than the freeform text I output now
* For the couple of stations I have that are broadcasting full PSIP, I see show title (from EIT) and description (from ETT), but nowhere do I see the episode title (subtitle in MythTV parlance). Not sure if it's not there, or I'm not parsing it from somewhere.

Please, give this program a try on your local stations and report back experiences. I'm especially interested in comparisons to other HDTV tuners that can display the guide info.

-Doug
View user's profile Send private message AIM Address
Re: New program: psipguide
PostPosted: Fri Mar 05, 2004 2:01 am Reply with quote
Ulmo
 
Joined: 06 Nov 2003
Posts: 95
Location: Aptos,CA,USA




dlarrick wrote:
* I am not processing CRCs in this code. Eventually it's bound for MythTV, which has routines to deal with the CRC, so this will happen at that time. But as a standalone program, it could use CRC.
* I have hardcoded the leap-second offset for GPS time; this should be parsed out of the System Time Table instead


You could have made me quieter here by simply looking at my hacks to your old testinfo.C playground that did just this, but I guess you figured it would be easy for me to just put them into your program again since I already did it once before, and I guess you're right.

I'll try to throw in CRC support and the GPS parsing a bit.. Edit update: ok, done; my modified version of your psipguide program included as part of my tools package; see:

ftp://ftp.sonic.net/pub/users/ulmo/hd2000/tools/

There is definitely a bug about gettting all the EITs. It often just quits, even though there's plenty of guide data that it missed. I might have really messed it up by returning true for ParseSTT, too, since I don't know what that return value means.

Also, Doug, here is some code I started writing. Mostly it's just tables, but you might find them useful. See the tutorial, too, for some ideas, but I don't always agree with it. I have a lot of other docs I found during this process not there, but you can find them easily enough, so I left them on our Mac for now.

ftp://ftp.sonic.net/pub/users/ulmo/t/I/tv/atsc/psip/

psip.c, mostly. Ignore the CRC32 code; my implementation in testinfo.C and psipguide.C as modified by me in my tools tarball above is a good implementation (it uses Linux kernel version which has, AFAIK, endian-aware 32 bit table oriented 32 bit aligned checking; the only next step to try is 64 bit for 64 bit processors).

Quote:
Please, give this program a try on your local stations and report back experiences. I'm especially interested in comparisons to other HDTV tuners that can display the guide info.


So far, it does output some stuff! Cool. See above for bug. I also made some changes to output format; I hope it gives you some ideas you can consider.

I haven't yet entered those Huffman tables in; are you parsing it? I'd be happy to copy & paste the Huffman tables for you like the tutorial says from the standard, if you like. It's nice to see the description.

Still not getting anything for my channel 12. Channel 10 & 13 ok. See ftp://ftp.sonic.net/pub/users/ulmo/hd2000/tools/test/psipguide/ for test output.


Last edited by Ulmo on Sat Mar 06, 2004 6:36 am; edited 1 time in total
View user's profile Send private message Send e-mail
PostPosted: Fri Mar 05, 2004 5:51 am Reply with quote
dlarrick
 
Joined: 22 Sep 2003
Posts: 58
Location: Outside Boston, MA




Quote:
You could have made me quieter here by simply looking at my hacks to your old testinfo.C playground that did just this


Yeah, I know... I did take a look at that but never got around to it.

Quote:
There is definitely a bug about gettting all the EITs. It often just quits, even though there's plenty of guide data that it missed


I need to take a look at this. Sometimes it just exits before doing anything, too. What it does is keep a set of PIDs that it should keep an eye on... when it thinks it's got everything it needs from a PID it returns true from ParseTableType and removes it from the set. When the set is empty it's done. I thought what was happening was that lack of CRC was letting some packets through that didn't actually contribute data, but I won't know until I try it.

ParseSTT should definitely not set pid_processed, as the STT pid is in the PSIP base PID with the MGT, and if encountered first could cause it to miss the MGT entirely.

I'll take a look over your code and pull some of those changes... I have a religious objection to GOTO statements, so I won't pull the whole thing. I also like my method of assembling the GPS epoch better (less likely to get me confused about time zones), though getting the leap-second offset the right way is good.

I have yet to encounter a station that's Huffman-encoding the ETTs, so while it's probably a good idea to include the tables, I have no way of testing it.

-Doug
View user's profile Send private message AIM Address
PostPosted: Sat Mar 06, 2004 8:12 am Reply with quote
Ulmo
 
Joined: 06 Nov 2003
Posts: 95
Location: Aptos,CA,USA




Quote:
There is definitely a bug about gettting all the EITs. It often just quits, even though there's plenty of guide data that it missed


Quote:
I need to take a look at this. Sometimes it just exits before doing anything, too. What it does is keep a set of PIDs that it should keep an eye on... when it thinks it's got everything it needs from a PID it returns true from ParseTableType and removes it from the set. When the set is empty it's done. I thought what was happening was that lack of CRC was letting some packets through that didn't actually contribute data, but I won't know until I try it.


No. I don't think the code is structured the way the standard has depicted it to work. You can't make lots of assumptions, really. That's why I left that tutorial -- I think they play fast a loose sometimes --- but really recommend a long read for the standard. Someday I was going to do it. I need to find work first.

E.g., one of my channels was never getting descriptions, but when I put a change in that just made it wait for STT, it suddenly started going far enough to get them.

There are some cool ideas in the tutorial, like "registering" for tables and parsed output.

Quote:
ParseSTT should definitely not set pid_processed, as the STT pid is in the PSIP base PID with the MGT, and if encountered first could cause it to miss the MGT entirely.


Ok. I made it also require ParseSTT.

Quote:
I'll take a look over your code and pull some of those changes... I have a religious objection to GOTO statements, so I won't pull the whole thing.


I actually don't know how to program without gotos, so you'll have to figure that out somehow. (I learned programming with assembler.) Hmmm ... I left code in there that was pre-goto, but I never trust redundant code (it causes bugs and is one of the symptoms of anti-goto programmers).

Quote:
I also like my method of assembling the GPS epoch better (less likely to get me confused about time zones), though getting the leap-second offset the right way is good.


Once again, I'm a puritan. Until one knows exactly everything that's happening, it doesn't make sense to carry around pre-cooked data that needs to be uncooked then re-cooked to be used. It's best to have pure raw data, so that one can always do the right thing with it at the right place. Eventually one builds exactly the functions one needs, where needed, as small as can be, and without bugs. Otherwise, nothing ever works, and it's always buggy, bloated, confusing and just hell.

Confused? Why is anyone programming that can get confused? If it's complex, you make it as simple as possible, and no simpler. After that, you just have to figure it out, not keep damaging it. Do not "KISS"; instead, Einstein it. Otherwise, it will be buggy and uninteresting.

Quote:
I have yet to encounter a station that's Huffman-encoding the ETTs, so while it's probably a good idea to include the tables, I have no way of testing it.

-Doug



I'll upload a patch to my last version. I don't feel like signing it right now. Goodnight.
View user's profile Send private message Send e-mail
SE TX listings
PostPosted: Thu Jul 22, 2004 9:24 pm Reply with quote
inkling
 
Joined: 05 Feb 2004
Posts: 342




CBS, KHOU-DT is only one broadcasting PSIP near me, as far as I can tell from the psipguide.C dated may 27 2004 (so sez wget).

CBS is sending 12 hours of programming info, but apparently no more.

Looks pretty nice for CBS streams.

That's the good news. Now for the bad.

All the rest of the station streams go on wild PID hunt and never find anything.

I only capture raw files so it's getting whatever the card can see.

The primary failure is "Still need x PIDs", on the streams that can't find any PSIP info.

It will be good when all bcasters are using PSIP, so we can adjust programming to match latest (re-) scheduling by name only.

I noticed ABC (CH32) headers showing up on psipguide scan of stream captured from UPN (CH19). Odd, because they're 78 MHz apart. ABC streams show only a header and a hunt.

Bad reflection, crosstalk who knows?

Interesting photonic down-conversion though, if it really did get CH32 stuff from CH19. I'll have to get out my slide rule and look for harmonics.

I'm just guessing here, but i'd say it's 7th harmonic addition from 501 and 579 overlapping.
Just a wild guess.

Concrete buildings scattered around as photonic crystals for medium frequency radio? Hmm...

Cantenna, anyone?

:>
View user's profile Send private message
Re: SE TX listings
PostPosted: Fri Jul 23, 2004 3:59 am Reply with quote
dlarrick
 
Joined: 22 Sep 2003
Posts: 58
Location: Outside Boston, MA




inkling wrote:

All the rest of the station streams go on wild PID hunt and never find anything.

I only capture raw files so it's getting whatever the card can see.

The primary failure is "Still need x PIDs", on the streams that can't find any PSIP info.


It should time out once it's seen 1000 MGT repetitions and give you whatever it's seen up to that point.

If an ETT or EIT is mentioned in the MGT (printed out up top) but never sent by the station, you'll see the behavior you cited. I had a station that was doing this for a while. Also, it can take 5-10 minutes' worth of video for a complete set of PSIP data.

inkling wrote:

It will be good when all bcasters are using PSIP, so we can adjust programming to match latest (re-) scheduling by name only.


I figure the idle behavior of Myth should be to spin through the stations with good PSIP, updating start/end times and wholesale replacements of shows. And of course, paying attention during the current recording and extending as required.

inkling wrote:

I noticed ABC (CH32) headers showing up on psipguide scan of stream captured from UPN (CH19). Odd, because they're 78 MHz apart. ABC streams show only a header and a hunt.


Could the two local stations be under the same management? If so, they could be doing this (intentionally or not). Are the listings correct? Smile
[/quote]
View user's profile Send private message AIM Address
Re: SE TX listings
PostPosted: Tue Jul 27, 2004 2:05 pm Reply with quote
inkling
 
Joined: 05 Feb 2004
Posts: 342




ABC is Disney. UPN is Viacom or FOX, not sure who wound up with KTXH.

They're not related, but the transmitters might be near each other. Most of them are in Missouri City.

There seems to still be some growing pains. Some stations will drop right off the air or have broken audio for hours or sometimes days.

It's pointless to tell them, they just go "duh?"

CBS seems only one sending, so I guess CBS is only one on the ball in Houston, but even they are only sending 12 hours. I sort of recall ATSC spec recommends 72 hours, but perhaps that is not feasible with HD + 5.1.

I wasn't really reporting a problem. I expect the spurious channels I saw were just that, spurious data from stream dropouts.

I know you mentioned it needed CRC, but didn't check to see if it was added on that version.

Streams fed to it were 30 to 120 minutes long.

Your program seems to work as long as the broadcaster is sending the data.

Keep up the good work!

The broadcasters will catch up eventually.
View user's profile Send private message
Re: SE TX listings
PostPosted: Wed Jul 28, 2004 7:45 pm Reply with quote
inkling
 
Joined: 05 Feb 2004
Posts: 342




>dlarrick:
>>inkling:
>>All the rest of the station streams go on wild PID hunt and never find anything.

I did some more testing. UPN finally does timeout after 1000 MGTs. I was impatient.

The rest of them return within seconds.

CBS works. 12 hours, some detail.

NBC event name is 'DTV Program' with all programs being 1800 seconds, and a 12 hour listing for each sub-channel.

ABC event name is same , but always start time is 19:00:00, 43200 seconds for each sub-channel.

WB doesn't show any events, 'Still need 2 PIDs', but does seem to show the sub-channel names.

FOX event name is 'KRIV1 1', start time is 19:00:00 10800 seconds.

UPN goes on a PID hunt and then finally bails and gives event 'KTXH1 1', also 19:00:00 10800 seconds.

>I figure the idle behavior of Myth should be to spin through the stations with good PSIP, updating start/end times and wholesale replacements of shows. And of course, paying attention during the current recording and extending as required.

I'd like to add a similar function to pchdtvr. Even user-initiated query would be nice. Can do the automatic stuff later. Have to dig for specs.

>>I noticed ABC (CH32) headers showing up on psipguide scan of stream captured from UPN (CH19). Odd, because they're 78 MHz apart. ABC streams show only a header and a hunt.

Checked NASA, space weather is a mess.

Took me an hour to find the UPN viewing position and that's usually the easiest one to find.

Can stuff the output of the various stations in http://www.nop.org/inkling/dtv/ if you like. they'll be *.psip or psip.tgz or whatever.

later
View user's profile Send private message
New program: psipguide
  pcHDTV Forum Index -> HDTV programming and listings
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
All times are GMT - 7 Hours  
Page 1 of 1  

  
  
 Post new topic  This topic is locked: you cannot edit posts or make replies.  


Powered by phpBB © 2001-2003 phpBB Group
Theme created by Vjacheslav Trushkin