Log in Register FAQ Memberlist Search pcHDTV Forum Index
pcHDTV Forum

pcHDTV Forum Index -> HDTV programming and listings -> XMLTV can now get HDTV listings from Zap2It.Com.
Post new topic  This topic is locked: you cannot edit posts or make replies. View previous topic :: View next topic 
XMLTV can now get HDTV listings from Zap2It.Com.
PostPosted: Sat Jan 24, 2004 1:16 am Reply with quote
Ulmo
 
Joined: 06 Nov 2003
Posts: 95
Location: Aptos,CA,USA




FYI, XMLTV has a "Hack to retrieving digital channel listings" http://mythtv.org/pipermail/mythtv-dev/2004-January/018217.html by Kirk Lapray <kirk@backfence.net> posted 2.5 days ago to MythTV-Dev mailing list http://mythtv.org/pipermail/mythtv-dev/.

I made a legible copy of his message if you just want to read the text: ftp://ftp.armory.com/pub/user/ulmo/HW/pcHDTV/HD-2000/apps/xmltv/read/body.txt.

I just tested it, and it works. See my notes http://mythtv.org/pipermail/mythtv-dev/2004-January/018304.html in response to his in that mailing list for anything else I had to say.

Since his message is a MIME message that's not MIME-parsed at the archive, I put the whole thing and its parts at ftp://ftp.armory.com/pub/user/ulmo/HW/pcHDTV/HD-2000/apps/xmltv/:
na_zap2it_kirk_lapray.patch.bz2 [whole RFC822 & MIME message]
message_body_wrapped.bz2 [text message]
zaplistings.diff.bz2 [diff]
cookie.txt.bz2 [cookie file used by patched pgm]
View user's profile Send private message Send e-mail
PostPosted: Sat Jan 24, 2004 1:07 pm Reply with quote
elwood
 
Joined: 26 Sep 2003
Posts: 67
Location: Phoenix, AZ




Good find. I saw that zap2it.com started putting the digital channels and subchannels for my area on their site, but not including them in the broadcast listings.

Since I pull both cable and broadcast listings, this hack doesn't work so well unless I run mythfilldatabase from two hosts.

My contention centers around xmltv only grabbing channels from its list, while I just want to add a handful more to a "custom" list.

But this is progress...

Brad
View user's profile Send private message
PostPosted: Sun Jan 25, 2004 6:40 pm Reply with quote
Ulmo
 
Joined: 06 Nov 2003
Posts: 95
Location: Aptos,CA,USA




elwood wrote:
Good find. I saw that zap2it.com started putting the digital channels and subchannels for my area on their site, but not including them in the broadcast listings.


For a while Zap2It.Com had it in the comments in the XML/HTML files they were serving, but had no actual listing data for them. Now, I found that some channels aren't listed, but when you go into the editor and delete some channels you don't want, you can replace them with a correctly specified channel call sign and channel # and it will actually use that , finding programs. I did that for our local channel 13 DT / 35 TV.

Quote:
Since I pull both cable and broadcast listings, this hack doesn't work so well unless I run mythfilldatabase from two hosts.


That's exactly how I got around that problem myself; really lazy, but it works. I think I did have to make sure my "mythfilldatabase" programs didn't conflict: I don't know what matters, but I just got paranoid and had them each manage exclusive frequency tables (one for broadcast and one for cable), and made sure they were both the same version as my backend (I compile Myth seperately on my hosts).

Quote:
My contention centers around xmltv only grabbing channels from its list, while I just want to add a handful more to a "custom" list.


There are about a dozen things this could mean: the custom list at Zap2It, which sucks in lots of ways that TitanTV's list does not (only one source at once at Zap2It, but Titan has multiple sources) (Am I the only one who is more confused by Titan's new format?); the seperate mythfilldatabases we run; the seperate input lists we give them; the half a dozen or so fields in the mythconverg database.


But yes, this is progress.


I almost forgot: I noticed a big problem; I think mythfilldatabase stuck duplicate slightly different channels in the channel database, and only has programs for one. I'll check right now.

Code:
mysql> SELECT chanid,channum,freqid,callsign,name,finetune FROM channel WHERE sourceid=1;
+--------+---------+--------+----------+--------------+----------+
| chanid | channum | freqid | callsign | name         | finetune |
+--------+---------+--------+----------+--------------+----------+
|   1008 | 8       | 10-1   | KSBWDT   | 10-1 KSBW-DT |        0 |
|   1011 | 11      | 12-4   | KNTVHD   | 12-4 KNTV-HD |        0 |
|   1035 | 35      | 13-2   | KCBADT   | 13-2 KCBA-DT |        0 |
|   1125 | 11      | 12-5   | KNTVDT   | 12-5 KNTV-SD |        0 |
|   1009 | 8       | 8      | KSBWDT   | 8 KSBWDT     |        0 |
|   1012 | 11      | 11     | KNTVDT   | 11 KNTVDT    |        0 |
|   1036 | 35      | 35     | KCBADT   | 35 KCBADT    |        0 |
+--------+---------+--------+----------+--------------+----------+
7 rows in set (0.00 sec)

mysql>


Uhoh. Those last three are not the ones I had wanted there. Which ones are getting programs?

Code:
SELECT chanid,starttime FROM program WHERE chanid<2000;
sort
 
chanid  starttime
1008    2004-01-24 00:37:00     The correct channel I originally set.
        [...]
        2004-01-31 23:30:00     Last entry.
 
1009    2004-01-24 00:37:00     The incorrect channel it made.
        [...]
        2004-02-02 23:35:00     Last entry.  This is the new data it's getting.
 
1011    2004-01-24 00:37:00     orig
        2004-01-31 23:30:00
1012    2004-01-24 00:37:00     dup
        2004-02-02 23:35:00
1035    2004-01-24 00:05:00     orig
        2004-01-31 23:00:00
1036    2004-01-24 00:05:00     dup
        2004-02-02 23:35:00


As you can see, the dups are the ones getting the new data.

The only fields different between the currently working ones and the original ones are channum, freqid, and name (and obviously chanid). Just for testing, I first slowly migrate the new ones to the old one's values:

Code:
mysql> SELECT chanid,channum,freqid,callsign,name,finetune FROM channel WHERE sourceid=1;
+--------+---------+--------+----------+--------------+----------+
| chanid | channum | freqid | callsign | name         | finetune |
+--------+---------+--------+----------+--------------+----------+
|   1008 | 8       | 10-1   | KSBWDT   | 10-1 KSBW-DT |        0 |
|   1011 | 11      | 12-4   | KNTVHD   | 12-4 KNTV-HD |        0 |
|   1035 | 35      | 13-2   | KCBADT   | 13-2 KCBA-DT |        0 |
|   1125 | 11      | 12-5   | KNTVDT   | 12-5 KNTV-SD |        0 |
|   1009 | 8       | 8      | KSBWDT   | 8 KSBWDT     |        0 |
|   1012 | 11      | 11     | KNTVDT   | 11 KNTVDT    |        0 |
|   1036 | 35      | 35     | KCBADT   | 35 KCBADT    |        0 |
+--------+---------+--------+----------+--------------+----------+
7 rows in set (0.00 sec)

mysql> mysql> UPDATE channel SET freqid='10-1' WHERE chanid=1009;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> UPDATE channel SET freqid='12-4' WHERE chanid=1012;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> UPDATE channel SET freqid='13-2' WHERE chanid=1036;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> SELECT chanid,channum,freqid,callsign,name,finetune FROM channel WHERE sourceid=1;
+--------+---------+--------+----------+--------------+----------+
| chanid | channum | freqid | callsign | name         | finetune |
+--------+---------+--------+----------+--------------+----------+
|   1008 | 8       | 10-1   | KSBWDT   | 10-1 KSBW-DT |        0 |
|   1011 | 11      | 12-4   | KNTVHD   | 12-4 KNTV-HD |        0 |
|   1035 | 35      | 13-2   | KCBADT   | 13-2 KCBA-DT |        0 |
|   1125 | 11      | 12-5   | KNTVDT   | 12-5 KNTV-SD |        0 |
|   1009 | 8       | 10-1   | KSBWDT   | 8 KSBWDT     |        0 |
|   1012 | 11      | 12-4   | KNTVDT   | 11 KNTVDT    |        0 |
|   1036 | 35      | 13-2   | KCBADT   | 35 KCBADT    |        0 |
+--------+---------+--------+----------+--------------+----------+
7 rows in set (0.00 sec)


While doing that, I was thinking we could write a script that converts HD-2000's dtvscan's output for Xine's ~/.xine/chanlist format into XMLTV channel list format AND a companion script that inserts the correct lines (with all information represented similarly to above) for Myth's mythconverg.channel table, AND another script that runs both of those correctly, with instructions to go edit it after it has done that, too.

Anyway, running mythfilldatabase for that channel range now:

here's a /tmp file it made during processing:
Code:
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE tv SYSTEM "xmltv.dtd">

<tv date="20040125055730 -0800" source-info-url="http://www.zap2it.com" source-i
nfo-name="Zap2It" generator-info-name="tv_grab_na V3.20031206" generator-info-ur
l="http://sourceforge.net/projects/xmltv">
  <channel id="C8ksbwdt.zap2it.com">
    <display-name>8 KSBWDT</display-name>
    <display-name>8</display-name>
  </channel>
  <channel id="C11kntvdt.zap2it.com">
    <display-name>11 KNTVDT</display-name>
    <display-name>11</display-name>
  </channel>
  <channel id="C35kcbadt.zap2it.com">
    <display-name>35 KCBADT</display-name>
    <display-name>35</display-name>
  </channel>


relevent output from run:
Code:
Day 2004-1-26 schedule for station 8 KSBWDT has:28 programs
Day 2004-1-26 schedule for station 11 KNTVDT has:28 programs
Day 2004-1-26 schedule for station 35 KCBADT has:38 programs

[...]

Data is already present for Sat Jan 31 2004, skipping
Data is already present for Sun Feb 1 2004, skipping
Data is already present for Mon Feb 2 2004, skipping


From what we know from above that we already had data in the new ones it made for Feb 2 but not the old ones, we know changing freqid did not change which channels it's updating. Next field to try:

Code:
mysql> UPDATE channel SET name='10-1 KSBW-DT' WHERE chanid=1009;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> UPDATE channel SET name='12-4 KNTV-HD' WHERE chanid=1012;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> UPDATE channel SET name='13-2 KCBA-DT' WHERE chanid=1036;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> SELECT chanid,channum,freqid,callsign,name,tvformat FROM channel WHERE sourceid=1;
+--------+---------+--------+----------+--------------+----------+
| chanid | channum | freqid | callsign | name         | tvformat |
+--------+---------+--------+----------+--------------+----------+
|   1008 | 8       | 10-1   | KSBWDT   | 10-1 KSBW-DT | ATSC     |
|   1011 | 11      | 12-4   | KNTVHD   | 12-4 KNTV-HD | ATSC     |
|   1035 | 35      | 13-2   | KCBADT   | 13-2 KCBA-DT | ATSC     |
|   1125 | 11      | 12-5   | KNTVDT   | 12-5 KNTV-SD | ATSC     |
|   1009 | 8       | 10-1   | KSBWDT   | 10-1 KSBW-DT |          |
|   1012 | 11      | 12-4   | KNTVDT   | 12-4 KNTV-HD |          |
|   1036 | 35      | 13-2   | KCBADT   | 13-2 KCBA-DT |          |
+--------+---------+--------+----------+--------------+----------+
7 rows in set (0.00 sec)

mysql> UPDATE channel SET tvformat='ATSC' WHERE sourceid=1;
Query OK, 3 rows affected (0.00 sec)
Rows matched: 7  Changed: 3  Warnings: 0

mysql>
$ mythfilldatabase
Day 2004-1-26 schedule for station 8 KSBWDT has:28 programs
Day 2004-1-26 schedule for station 11 KNTVDT has:28 programs
Day 2004-1-26 schedule for station 35 KCBADT has:38 programs

Data is already present for Sun Feb 1 2004, skipping
Data is already present for Mon Feb 2 2004, skipping

mysql> SELECT chanid,channum,freqid,callsign,name,tvformat FROM channel WHERE sourceid=1;
+--------+---------+--------+----------+--------------+----------+
| chanid | channum | freqid | callsign | name         | tvformat |
+--------+---------+--------+----------+--------------+----------+
|   1008 | 8       | 10-1   | KSBWDT   | 10-1 KSBW-DT | ATSC     |
|   1011 | 11      | 12-4   | KNTVHD   | 12-4 KNTV-HD | ATSC     |
|   1035 | 35      | 13-2   | KCBADT   | 13-2 KCBA-DT | ATSC     |
|   1125 | 11      | 12-5   | KNTVDT   | 12-5 KNTV-SD | ATSC     |
|   1009 | 8       | 10-1   | KSBWDT   | 10-1 KSBW-DT | ATSC     |
|   1012 | 11      | 12-4   | KNTVDT   | 12-4 KNTV-HD | ATSC     |
|   1036 | 35      | 13-2   | KCBADT   | 13-2 KCBA-DT | ATSC     |
+--------+---------+--------+----------+--------------+----------+
7 rows in set (0.00 sec)

mysql>


While that was running, I looked at what the real difference was: I used to use NTSC listings to set the listings for those ATSC channels (which worked quite well), but look at xmltvid:

Code:
mysql> SELECT chanid,name,icon,xmltvid FROM channel WHERE tvformat='ATSC';
+--------+--------------+------------------------------------------+----------------------+
| chanid | name         | icon                                     | xmltvid              |
+--------+--------------+------------------------------------------+----------------------+
|   1008 | 10-1 KSBW-DT | /home/mythbe/.mythtv/channels/nbc_30.jpg | C8ksbw.zap2it.com    |
|   1011 | 12-4 KNTV-HD | /home/mythbe/.mythtv/channels/nbc_30.jpg | C11kntv.zap2it.com   |
|   1035 | 13-2 KCBA-DT | /home/mythbe/.mythtv/channels/fox_30.jpg | C35kcba.zap2it.com   |
|   1125 | 12-5 KNTV-SD | /home/mythbe/.mythtv/channels/nbc_30.jpg | C11kntv.zap2it.com   |
|   1009 | 10-1 KSBW-DT |                                          | C8ksbwdt.zap2it.com  |
|   1012 | 12-4 KNTV-HD |                                          | C11kntvdt.zap2it.com |
|   1036 | 13-2 KCBA-DT |                                          | C35kcbadt.zap2it.com |
+--------+--------------+------------------------------------------+----------------------+
7 rows in set (0.00 sec)

mysql>

...
mysql> SELECT chanid,name,xmltvid,recpriority FROM channel WHERE tvformat='ATSC';
+--------+--------------+----------------------+-------------+
| chanid | name         | xmltvid              | recpriority |
+--------+--------------+----------------------+-------------+
|   1008 | 10-1 KSBW-DT | C8ksbwdt.zap2it.com  |           6 |
|   1011 | 12-4 KNTV-HD | C11kntvdt.zap2it.com |           8 |
|   1035 | 13-2 KCBA-DT | C35kcbadt.zap2it.com |           9 |
|   1125 | 12-5 KNTV-SD | C11kntvdt.zap2it.com |           2 |
|   1009 | 10-1 KSBW-DT | C8ksbwdt.zap2it.com  |           0 |
|   1012 | 12-4 KNTV-HD | C11kntvdt.zap2it.com |           0 |
|   1036 | 13-2 KCBA-DT | C35kcbadt.zap2it.com |           0 |
+--------+--------------+----------------------+-------------+
7 rows in set (0.00 sec)

mythfilldatabase:
Data is already present for Sat Jan 31 2004, skipping
Data is already present for Sun Feb 1 2004, skipping
Data is already present for Mon Feb 2 2004, skipping


Well, no, the ones I fixed don't have Feb 1, 2 yet. Let's see in program. Yes, the new ones are only in the new channel entries.

I deleting the new entries in program and channel; here's the new channel status:

Code:
$ echo "SELECT * FROM channel WHERE tvformat='ATSC';"|ms -B |sort -n
chanid  channum freqid  sourceid        callsign        name    icon    finetune        videofilters   xmltvid recpriority     contrast        brightness      colour  hue     tvformat
1008    8       10-1    1       KSBWDT  10-1 KSBW-DT            0       NULL    C8ksbwdt.zap2it.com    6       32768   32768   32768   32768   ATSC
1011    11      12-4    1       KNTVHD  12-4 KNTV-HD            0       NULL    C11kntvdt.zap2it.com   8       32768   32768   32768   32768   ATSC
1035    35      13-2    1       KCBADT  13-2 KCBA-DT            0       NULL    C35kcbadt.zap2it.com   9       32768   32768   32768   32768   ATSC
1125    11      12-5    1       KNTVDT  12-5 KNTV-SD    /home/mythbe/.mythtv/channels/nbc_30.jpg       0       NULL    C11kntvdt.zap2it.com    2       32768   32768   32768   32768   ATSC
$


Trying mythfilldatabase again:

Code:
Day 2004-1-26 schedule for station 8 KSBWDT has:28 programs
Day 2004-1-26 schedule for station 11 KNTVDT has:28 programs
Day 2004-1-26 schedule for station 35 KCBADT has:38 programs


Data is already present for Thu Jan 29 2004, skipping
Data is already present for Fri Jan 30 2004, skipping
Data is already present for Sat Jan 31 2004, skipping
Day 2004-2-1 schedule for station 8 KSBWDT has:29 programs
Day 2004-2-1 schedule for station 11 KNTVDT has:28 programs
Day 2004-2-1 schedule for station 35 KCBADT has:23 programs


So, let's see if that actually means it worked:
Code:
$ echo "SELECT * FROM channel WHERE tvformat='ATSC';"|ms -B |sort -n
chanid  channum freqid  sourceid        callsign        name    icon    finetune        videofilters   xmltvid recpriority     contrast        brightness      colour  hue     tvformat
1008    8       10-1    1       KSBWDT  10-1 KSBW-DT            0       NULL    C8ksbwdt.zap2it.com    6       32768   32768   32768   32768   ATSC
1011    11      12-4    1       KNTVHD  12-4 KNTV-HD            0       NULL    C11kntvdt.zap2it.com   8       32768   32768   32768   32768   ATSC
1035    35      13-2    1       KCBADT  13-2 KCBA-DT            0       NULL    C35kcbadt.zap2it.com   9       32768   32768   32768   32768   ATSC
1125    11      12-5    1       KNTVDT  12-5 KNTV-SD    /home/mythbe/.mythtv/channels/nbc_30.jpg       0       NULL    C11kntvdt.zap2it.com    2       32768   32768   32768   32768   ATSC
$

good ...

$ echo "SELECT starttime FROM program WHERE chanid<2000;"|ms -B |sort -n |head -3
starttime
20040124000500
20040124003500
$ echo "SELECT starttime FROM program WHERE chanid<2000;"|ms -B |sort -n |tail -2
20040202233500
20040202233500
$


GREAT! Ok, so having exactly duplicate lines in the database doesn't mean they all get populated, but I think the problem I had before is that I did not update xmltvid.
View user's profile Send private message Send e-mail
XMLTV can now get HDTV listings from Zap2It.Com.
  pcHDTV Forum Index -> HDTV programming and listings
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
All times are GMT - 7 Hours  
Page 1 of 1  

  
  
 Post new topic  This topic is locked: you cannot edit posts or make replies.  


Powered by phpBB © 2001-2003 phpBB Group
Theme created by Vjacheslav Trushkin