March 2007 - EpiData-list - University of Manitoba Mailing Lists

Manipulation of Date end hour
by epidata-list＠lists.umanitoba.ca 13 Mar '07

13 Mar '07

I would like to know how to treat(manipulate) dates and hours at the same time. I have a data base which I seized under Epidata Entry and who understand variables dates and the hour of entrance for the delivery and date and the hour for the delivery. The hour I made two numeric variables, the hour and minutes. Now how to make to have the difference in the hour or in minutes? The problem does not settlewhen the delivery with place the same day. But the problem when there is gap For example: DAR Date of arrival <10/01/2007> HAR Arrival time 13 MAR Minute of arrival 45 DAC Dates delivery <12/01/2007> HAC Hour of delivery 02 MAC Minutes of delivery 34 Under SPSS, the manipulation is easy, I would like to know if under Epidata Analysis we can do the same with these temporal data. Should the opposite occur it would be necessary to think of introducing this module into the final version of this magnificent software. ___________________________________________ Dr Jean-Paul DIBY-KONAN Médecin Santé Publique MCM Santé de la Reproduction Institut Médecine Tropicale Kapucinessenstraat 8 B-2000 Anvers Belgique ----- Message d'origine ---- De : "epidata-list-request(a)lists.umanitoba.ca" <epidata-list-request(a)lists.umanitoba.ca> À : epidata-list(a)lists.umanitoba.ca Envoyé le : Mardi, 13 Mars 2007, 18h10mn 43s Objet : EpiData-list Digest, Vol 41, Issue 7 Send EpiData-list mailing list submissions to epidata-list(a)lists.umanitoba.ca To subscribe or unsubscribe via the World Wide Web, visit http://lists.umanitoba.ca/mailman/listinfo/epidata-list or, via email, send a message with subject or body 'help' to epidata-list-request(a)lists.umanitoba.ca You can reach the person managing the list at epidata-list-owner(a)lists.umanitoba.ca When replying, please edit your Subject line so it is more specific than "Re: Contents of EpiData-list digest..." EpiData-list mailing list ___________________________________ Today's Topics: 1. deleted and excluded records (epidata-list(a)lists.umanitoba.ca) 2. Re: deleted and excluded records (epidata-list(a)lists.umanitoba.ca) ---------------------------------------------------------------------- Message: 1 Date: Tue, 13 Mar 2007 08:16:27 +0100 (CET) From: epidata-list(a)lists.umanitoba.ca Subject: [EpiData-list] deleted and excluded records To: <epidata-list(a)lists.umanitoba.ca> Message-ID: <52289.130.226.78.129.1173770187.squirrel(a)webmail.webhosting.dk> Content-Type: text/plain; charset=iso-8859-1 Continued discussion on this topic: The suggested solution I gave could be clarified: 1. Dates before year approx. 1700-1800 give problems in Analysis now, but obviously should not. Although as everyone probably know there is a shift around 15xx (I do not remember the exact year). 2. The suggestion: "SUGGESTION: Would it not be possible for Analysis to assign a standard valid value (e.g. 01/01/1900) to any invalid dates it comes across during the read process. " Will be used in the next build of analysis, since it is very easy to implement. Obviously with a message to the user. I think this can be done in 5 minutes of work time in the programming, whereas the optimal solution of showing variable/field name and record is a large job - could be 1-2 days of work, since the function controlling the date value is not knowing neither the name of the field nor the record. But I will add the issue to the work plan for a solution at some point. Regarding the "excluded" versus "deleted" is also easy. The user can control whether records deleted are read or not by: set read deleted=on set read deleted=off The change works only when reading the file, so if you change re-read the file again. e.g. set read deleted = off read myfile describe age set read deleted = on read myfile /close describe age // here records marked for deletion are included. off is default. There is an error in v1 b68 in relation to this setting, such that if you read the file with "set read deleted=off" and then do an "update", you will see the records marked for deletion. The error has been found and a solution will be shown in next test release. regards Jens Lauritsen ------------------------------ Message: 2 Date: Tue, 13 Mar 2007 09:43:23 +0200 From: epidata-list(a)lists.umanitoba.ca Subject: Re: [EpiData-list] deleted and excluded records To: epidata-list(a)lists.umanitoba.ca Message-ID: <44e66f9b0703130043t1911363ev728eaea2e245ef85(a)mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Thanks for the quick response to my email. I would just like to clarify my suggestion re "excluded" and "deleted" records. The issue is not about whether one can control reading deleted records or not (as Jens' response implies). My point was that if one is not aware that records have been deleted, it would be more informative on reading the dataset in Analysis if the message specifically states that x records have been deleted (and are therefore excluded). Using the word "excluded" suggests that Analysis has a problem with some values and therefore has excluded the records. One could spend quite some time trying to identify the problem before realising that these are merely records that have been deleted. On the point of deleted records, is there some way in Analysis of identifying records which had been deleted (but which one has chosen to include in the analysis). In EpiInfo, there was a system variable RECDELETED. Also, in browse/update mode, deleted records were identifiable by an asterisk. This was very useful in trying to sort out why records had been deleted. Thanks again, Annemieke van Middelkoop ------------------------------ ________________________________________ EpiData-list(a)lists.umanitoba.ca http://lists.umanitoba.ca/mailman/listinfo/epidata-list End of EpiData-list Digest, Vol 41, Issue 7 ******************************************* ___________________________________________________________________________ Découvrez une nouvelle façon d'obtenir des réponses à toutes vos questions ! Profitez des connaissances, des opinions et des expériences des internautes sur Yahoo! Questions/Réponses http://fr.answers.yahoo.com

1 1

deleted and excluded records and dates
by epidata-list＠lists.umanitoba.ca 13 Mar '07

13 Mar '07

Hi all, Interesting disccussion here. > 2. The suggestion: > "SUGGESTION: Would it not be possible for Analysis to assign a standard > valid value (e.g. 01/01/1900) to any invalid dates it comes across > during the read process. " > > Will be used in the next build of analysis, since it is very easy to > implement. Obviously with a message to the user. I think this can be > done in 5 minutes of work time in the programming, whereas the optimal > solution of showing variable/field name and record is a large job - > could be 1-2 days of work, since the function controlling the date value > is not knowing neither the name of the field nor the record. But I will > add the issue to the work plan for a solution at some point. In my view the problem with that solution is that if Analysis automatically changes all "invalid" dates to any standard missing value, it can be a problem if the user saves the changes in a new data file and deletes the old one. The message for the users should be VERY CLEAR. > I would just like to clarify my suggestion re "excluded" and "deleted" > records. The issue is not about whether one can control reading > deleted records or not (as Jens' response implies). My point was that > if one is not aware that records have been deleted, it would be more > informative on reading the dataset in Analysis if the message > specifically states that x records have been deleted (and are > therefore excluded). Using the word "excluded" suggests that Analysis > has a problem with some values and therefore has excluded the records. > One could spend quite some time trying to identify the problem before > realising that these are merely records that have been deleted. Maybe we can use a text similar to the one used by EpiInfo for Windows: "#### records loaded (deleted records excluded/included)" Still it makes no senese to display 100 times the same message saying that a record was excluded because of an invalide date if the user can not do anything. In my view one solution can be display, in adition to the previous message, another message saying: "#### records excluded because of quality checking with dates(or other reasons). TIP: sort ascending your data in any date variable and browse them to detect the problems" Pedro Arias

1 0

deleted and excluded records
by epidata-list＠lists.umanitoba.ca 13 Mar '07

13 Mar '07

Continued discussion on this topic: The suggested solution I gave could be clarified: 1. Dates before year approx. 1700-1800 give problems in Analysis now, but obviously should not. Although as everyone probably know there is a shift around 15xx (I do not remember the exact year). 2. The suggestion: "SUGGESTION: Would it not be possible for Analysis to assign a standard valid value (e.g. 01/01/1900) to any invalid dates it comes across during the read process. " Will be used in the next build of analysis, since it is very easy to implement. Obviously with a message to the user. I think this can be done in 5 minutes of work time in the programming, whereas the optimal solution of showing variable/field name and record is a large job - could be 1-2 days of work, since the function controlling the date value is not knowing neither the name of the field nor the record. But I will add the issue to the work plan for a solution at some point. Regarding the "excluded" versus "deleted" is also easy. The user can control whether records deleted are read or not by: set read deleted=on set read deleted=off The change works only when reading the file, so if you change re-read the file again. e.g. set read deleted = off read myfile describe age set read deleted = on read myfile /close describe age // here records marked for deletion are included. off is default. There is an error in v1 b68 in relation to this setting, such that if you read the file with "set read deleted=off" and then do an "update", you will see the records marked for deletion. The error has been found and a solution will be shown in next test release. regards Jens Lauritsen

1 1

READ statement / Invalid dates / How to handle dates in analysis
by epidata-list＠lists.umanitoba.ca 12 Mar '07

12 Mar '07

READ statement / Invalid dates / How to handle dates in analysis 1. I submitted my previous email concerning the READ statement because: - on reading a particular file, a list of some 20+ invalid dates was produced - followed by a message that there were x valid records and 10 records excluded from analysis. I assumed that the 10 records were excluded because of the invalid dates. Subsequently, I have discovered that there was no connection between the invalid dates and the excluded records! In fact, the invalid records were records which had been deleted. SUGGESTION: When records are excluded because they are deleted records, could the message use the words "Deleted records" rather than "Excluded records". Just to avoid any confusion! 2. To respond to Jamie's email (Invalid dates), some of the invalid dates had been entered by EpiInfo, but about half had been entered using EpiData. EpiData Entry has no problems accepting a date such as 12/03/206. (I have also tried 12/03/0206 which just immediately reverts to 12/03/206). Yet EpiData Analysis sees such a date as invalid. SUGGESTION: EpiData Entry and Analysis should be consistent with their interpretation of invalid dates. 3. To respond to Jens' email (How to handle dates in analysis), illegal dates are not the issue I am addressing. Illegal dates are not accepted by EpiData and will therefore not be an issue in Analysis. I am addressing legal dates which may have been entered incorrectly, e.g. 12/03/2006 inadvertently entered as 12/03/206. EpiData accepts this date. But Analysis does not accept it as a valid date. 4. Once the dataset with invalid dates has been read in Analysis, it is not possible to carry out any commands which might include such dates. BROWSE, LIST, SAVEDATA for example will not operate (unless one specifies variables excluding those which have invalid values). SUGGESTION: Would it not be possible for Analysis to assign a standard valid value (e.g. 01/01/1900) to any invalid dates it comes across during the read process. A message to this effect could be added to the list of invalid dates (with their variable names and record numbers). It is then up to the user to edit the invalid dates. But then at least Analysis will be able to carry out all commands. Many thanks, Annemieke van Middelkoop Pretoria, South Africa

1 0

How to handle dates in analysis
by epidata-list＠lists.umanitoba.ca 11 Mar '07

11 Mar '07

Clarification: Neither EpiData Entry nor EpiData Analysis can have date fields with illegal dates. E.g. February 29th is only allowed in leap years. Epi6 could save illegal dates and never controlled the data while reading for correct content. >From the beginning of EpiData all date fields were included as "true dates", meaning that there cannot be a date value which does not exist in the past or future. (dates before year 1600 can give problems) For EpiData Analysis this has given some users problems since it was difficult to see in which records the date problems were located if some dates were rejected. Also some old dates before year 1800 have not been accomodated fully in analysis, but this should be a minor problem, since we never or very rarely have data from that period. And if we do one could split the dates into day month and year parts as three variables, although this would remove the automatic validation of the content of the data. I suggest that we look at the possibility of implementing: > c. That, if neither a. nor b. is an option, more detail about the > 'invalid' dates (record number, variable name) is provided so that > they can be edited. since we cannot use an illegal date for anything. Analysis with incorrect dates makes no sense. Once we find a workable principle for this the same technical principle and list can include also names of variables/fields, which will be in conflict with internal functions, e.g. year, date or tab. -- Jens Lauritsen Coordinator and initiator of EpiData Project http://www.epidata.dk

1 0

Learning material in Portuguese
by epidata-list＠lists.umanitoba.ca 10 Mar '07

10 Mar '07

I added a link to the learning material in Portuguese on the Download page (www.epidata.dk/download.php) - thanks to the group in Rio Janeiro. It is good if users start experimenting with development of "WIKI" based notes and materials. You can do so by at the address www.epidata.org/wiki After some experimentation period and establishment of a responsible group for the Wiki a separation can be made btw. "official" materials and materials at the responsibility of users. regards Jens Lauritsen EpiData Association

1 0

INVALID dates
by epidata-list＠lists.umanitoba.ca 10 Mar '07

10 Mar '07

Dear Friend, Epi Analysis is a sensitive tool to detect the invalid dates that can be corrected. Of course these invalid dated can be prevented altogether by inserting date ranges in the check files also before data entry. Epi Analysis also detects the corrupted records in the file while READing it, which EPI6 was not able to do. Hence screening of dataset while loading is an important for data cleaning also and should be retained. I fully endorse the option c. suggested by you. Just like providing record numbers of corrupted records, the record with invalid dates should be listed with record number and if possible the date variable name also. With regards, Dr. Shavinder Singh

1 0

Re: [EpiData-list] invalid dates
by epidata-list＠lists.umanitoba.ca 09 Mar '07

09 Mar '07

While a solution that flags dates as invalid/missing will be useful, another strategy for dealing with the invalid dates is to import them as text and use the built-in functions to convert valid ones to dates. There is still a potential problem with finding the invalid date records, but you have an opportunity to do this on your own terms. I assume that the dates were imported somehow into a .rec file, as entry would not allow such dates. I don't recall whether import checks date validity, but it does other field types. The invalid dates can readily be checked using the validation capacity of EpiData Entry and .chk files. This may be the best way to do this, as a detailed report of errors is produced. I have run into this type of error importing EpiInfo files and reading .dbf files, but not working with .rec files that I had control over. Jamie > > I have several concerns about this approach. > 1. Most important, records with 'invalid' dates are excluded from > analysis (10 in this example). The fact that the 'invalid' dates are > likely to be incorrect is no reason to not be able to analyse the > records. > 2. READing a large dataset with many dates can be a very slow process. > 3. The output given by Analysis (as shown above) is not very useful. > Neither record numbers nor variable names are identified. > > I would like to suggest: > a. That dates are not checked at all during the READ process. The > analyst can look for incorrect dates in Analysis, relevant to the > content of the data. > OR > b. That the user can specify whether to check the dates or not on > READing a dataset. > OR > c. That, if neither a. nor b. is an option, more detail about the > 'invalid' dates (record number, variable name) is provided so that > they can be edited. > > Annemieke van Middelkoop > Pretoria, South Africa >

1 0

Learning material
by epidata-list＠lists.umanitoba.ca 09 Mar '07

09 Mar '07

Hello all, Sometime a go, I tried to find out some learning/teaching material for EpiData entry in portuguese. Unfortunately, I didnt find any. Since this year is the first year that I'll teach a class about computer program for clinical research to pos-graduation students, I decided to make a small material in portuguese. A sort of introductory step by step guide for beginners and some little more complex examples. If any would like to take a look, it is available at : http://www.ipec.fiocruz.br/materialdidatico/Aplicativos_pesquisa/2007/EpiDa… By -- Abraço forte e que a força esteja com você, Pedro Emmanuel

1 0

RE: EpiData-list Digest, Vol 41, Issue 2
by epidata-list＠lists.umanitoba.ca 09 Mar '07

09 Mar '07

I agree with the approach of Annenmieke van Middelkoop about 'invalid' records after "Read". Bernard Branger, Nantes, France __________________________________________________________________________ I have several concerns about this approach. 1. Most important, records with 'invalid' dates are excluded from analysis (10 in this example). The fact that the 'invalid' dates are likely to be incorrect is no reason to not be able to analyse the records. 2. READing a large dataset with many dates can be a very slow process. 3. The output given by Analysis (as shown above) is not very useful. Neither record numbers nor variable names are identified. I would like to suggest: a. That dates are not checked at all during the READ process. The analyst can look for incorrect dates in Analysis, relevant to the content of the data. OR b. That the user can specify whether to check the dates or not on READing a dataset. OR c. That, if neither a. nor b. is an option, more detail about the 'invalid' dates (record number, variable name) is provided so that they can be edited. Annemieke van Middelkoop Pretoria, South Africa ------------------------------ ________________________________________ EpiData-list(a)lists.umanitoba.ca http://lists.umanitoba.ca/mailman/listinfo/epidata-list End of EpiData-list Digest, Vol 41, Issue 2 *******************************************

1 0